From 76bb37eaed11bede2a44468be421e4a411015fba Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Wed, 2 Sep 2026 14:51:55 +0200 Subject: [PATCH 01/11] chore: re-capture the projection fixture corpus from pipelex dev (#58) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore: re-capture the projection fixture corpus from pipelex dev The shared inputs-template projection corpus moved in `pipelex`, so the copy committed here was stale. This is a pure re-capture: no projection code changed, because the Python projection reads whatever kind the descriptor states and both moves are descriptor-side. The capture adds one pipe, `input_semantics_scaffold.scaffold_open_natives`, which puts the natives whose payload shape the descriptor states openly at slot positions — the one place the corpus never reached. `native.Dynamic` and `native.Composite` are structureless by design, so their node is `unknown` and both renderers agree on the empty mapping. `native.JSON` is not: its pinned blueprint carries a required `json_obj`, which makes the dict inside it the corpus's first required unknown node. That position is what the capture declares as a new divergence class, `unknown-empty-object`. The projection renders the empty object, because the descriptor withholds the payload shape there and inventing one would stop it projecting the descriptor; the reference engine reflects the runtime content class instead and fills the dict with a sample key/value pair whoever fills the template in has to delete. The empty object round-trips through the input shaper cleanly, so the class records a difference of vantage rather than a defect and carries no ledger item. `input_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with the new pipe; every pre-existing key in all three is byte-for-byte unchanged. `mthds-js` commits the identical bytes. Advances L-260831-56a78f Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TEH4cihpjsEwgtKL16nzbK * docs: hold the divergence record's prose half to the manifest Review follow-ups on the open-natives recapture, all outside the capture itself — the fixture bytes are untouched and stay byte-identical to the mthds-js twin. The README beside the corpus explains each divergence class in prose, and that was the one half of the record nothing enforced: the manifest is checked hard, so a class that stops occurring has to be retired deliberately, but a retired class used to leave its bullet behind describing a difference that no longer exists, with the suite still green. `test_inputs_template_projection.py` now holds the README's bullet list to the manifest's own declarations, in both directions — the twin of the check `mthds-js` carries. `input_form.json` and `pipe_io_contracts.json` are one capture taken in one command, but nothing outside the contracts file's own parse test read it, so one half could lose a pipe the other still held with the whole suite green. Deleting the new pipe's contract took tests/unit from 557 passed to 556 passed and no failures; the two payload files now have to name the same pipes. The changelog entry claimed the JSON blueprint's `json_obj` is the corpus's first required unknown node. It is not: `widget.matrix[]`, `opt.matrix[]` and `many[].matrix[]` are required unknown nodes and pre-date this capture. What is new is the position — the first required unknown node at an object-field rather than a list-item slot, which is where the engine's rendering diverges. Two more prose claims the committed bytes contradict: the README's summary of which divergence entries carry a `null` ledger item named two of the three there now, and it still described byte parity as skip-gated on a projection that has since landed under L-260830-e7c5b5, so the job runs. Advances L-260831-56a78f Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TEH4cihpjsEwgtKL16nzbK --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 8 +++ tests/fixtures/protocol/README.md | 5 +- tests/fixtures/protocol/input_form.json | 39 +++++++++++++ ...caffold.scaffold_open_natives.compact.json | 16 ++++++ ...caffold.scaffold_open_natives.compact.toml | 6 ++ ...affold.scaffold_open_natives.explicit.json | 16 ++++++ ...affold.scaffold_open_natives.explicit.toml | 14 +++++ .../protocol/inputs_template/manifest.json | 29 +++++++++- .../fixtures/protocol/pipe_io_contracts.json | 57 +++++++++++++++++++ tests/unit/test_inputs_template_projection.py | 21 +++++++ 10 files changed, 208 insertions(+), 3 deletions(-) create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.toml diff --git a/CHANGELOG.md b/CHANGELOG.md index f2f99ff..dd2f717 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## [Unreleased] + +### Changed + +- **The projection fixture corpus covers the natives whose payload shape the descriptor states openly.** A new capture from `pipelex` `dev` adds `input_semantics_scaffold.scaffold_open_natives`, a pipe putting `native.JSON`, `native.Dynamic` and `native.Composite` each at a slot position — the one place the corpus never reached. `Dynamic` and `Composite` are structureless by design, so their node is `unknown` and both renderers agree on the empty mapping; `JSON` is not, because its pinned blueprint carries a required `json_obj`, which puts an unknown node at a **required object-field** position for the first time in the corpus — the required unknown nodes it already held sit at list-item positions, where no divergence was declared. That position is what the capture declares as a new divergence class, `unknown-empty-object`: the projection renders the empty object, because the descriptor withholds the payload shape there and inventing one would stop it projecting the descriptor, while the reference engine reflects the runtime content class and fills the dict with a sample key/value pair whoever fills the template in has to delete. The empty object round-trips through the input shaper cleanly, so the class records a difference of vantage rather than a defect and carries no ledger item. `input_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with it, and `mthds-js` commits the identical bytes. + +- **The corpus suite notices a capture that arrives half-formed.** Two properties the record always had were never asserted, so either could break with the suite still green. The README beside the corpus explains each divergence class in prose, and that was the half nothing enforced: the manifest is checked hard, so a class that stops occurring has to be retired deliberately, but a retired class could leave its bullet behind describing a difference that no longer exists — and the suite, parametrised over the manifest's own list, would simply shrink rather than fail. `test_inputs_template_projection.py` now holds the README's bullet list to the manifest's declarations in both directions, the twin of the check `mthds-js` carries. It also asserts that `input_form.json` and `pipe_io_contracts.json` name the same pipes: they are one capture taken in one command, but nothing outside the contracts file's own parse test read it, so one half could lose a pipe the other still held. + ## [v0.12.0] - 2026-09-01 ### Added diff --git a/tests/fixtures/protocol/README.md b/tests/fixtures/protocol/README.md index 8010875..ca984a8 100644 --- a/tests/fixtures/protocol/README.md +++ b/tests/fixtures/protocol/README.md @@ -25,7 +25,7 @@ One file per pipe, shape and format: `..`, holding the projection's exact return value (the JSON carries no trailing newline; the TOML carries exactly one). `inputs_template/manifest.json` states what the corpus covers, where it departs from the reference engine's own inputs-template renderer, and which of its own templates the runtime's input shaper refuses today. -The corpus is **contract-first**: it landed before either projection, so each is written against a stated expectation instead of the expectation being back-filled from whatever the first implementation happened to emit. `tests/unit/test_inputs_template_projection.py` runs its jobs — byte parity (skip-gated until `mthds.protocol.inputs_template` exists, `L-260830-e7c5b5`), kind coverage against the whole closed `FieldKind` vocabulary, file-set completeness, the divergence lapse check, and the integrity of the unshapeable record. +The corpus is **contract-first**: it landed before either projection, so each is written against a stated expectation instead of the expectation being back-filled from whatever the first implementation happened to emit. `tests/unit/test_inputs_template_projection.py` runs its jobs — byte parity (skip-gated on `mthds.protocol.inputs_template`; that projection has landed under `L-260830-e7c5b5`, so the job runs), kind coverage against the whole closed `FieldKind` vocabulary, file-set completeness, the agreement of the two payload files on which pipes the capture holds, the divergence lapse check, the agreement of the bullet list below with the manifest's own declarations, and the integrity of the unshapeable record. ### Why it departs from the engine, deliberately @@ -36,8 +36,9 @@ The expectation is not the engine's output. It is authored by a reference projec - `fixed-count-honoured` — a `Concept[N]` slot renders N elements; the engine emits one, which the runtime's own input shaper then rejects, so its template does not run. - `text-named-url` — a text field merely **named** `url` takes a text placeholder; the engine picks a placeholder by field name. - `object-native-keeps-envelope` — a native that renders as an object once its optional field is included keeps its `{concept, content}` envelope, because the shaper dispatches a native's bare value on its scalar kind and would reject the bare object. The engine unwraps to a scalar, which it can only do because it drops the optional field. A consequence of the first entry. +- `unknown-empty-object` — an `unknown` node renders as the empty object, because the descriptor withholds the payload shape at that position and a projection that invented one would have stopped projecting the descriptor. The engine reflects the runtime content class instead and fills a required dict with a sample key/value pair whoever fills the template in then has to delete. The empty object round-trips through the shaper cleanly, so leaving it empty costs nothing. -Each entry carries the workspace-ledger item tracking the engine fix, or `null` where the difference is one of vantage rather than a defect — the file-leaf entry is the descriptor's vantage, and the object-native one is a consequence of the optional-field entry rather than its own bug. +Each entry carries the workspace-ledger item tracking the engine fix, or `null` where the difference is one of vantage rather than a defect — the file-leaf and unknown-empty entries are the descriptor's vantage, and the object-native one is a consequence of the optional-field entry rather than its own bug. The templates are held to a second bar that is easy to lose sight of behind the byte parity: they must still **run**. A template is what someone fills in and hands back, so every slot of it has to survive the runtime's own input shaper. That is what separates a deliberate divergence from a projection bug — the file-leaf entry pins `{"url": ...}` because the shaper accepts exactly that wrapper, and the object-native entry keeps the envelope for the same reason. Where a class is retired, the corpus is regenerated and its entry disappears from the manifest on its own. diff --git a/tests/fixtures/protocol/input_form.json b/tests/fixtures/protocol/input_form.json index e818145..6f8fb5f 100644 --- a/tests/fixtures/protocol/input_form.json +++ b/tests/fixtures/protocol/input_form.json @@ -1444,5 +1444,44 @@ } } ] + }, + "input_semantics_scaffold.scaffold_open_natives": { + "fields": [ + { + "kind": "object", + "name": "json_in", + "concept_ref": "native.JSON", + "description": "A JSON object", + "required": true, + "presence": "plain", + "gating": true, + "fields": [ + { + "kind": "unknown", + "name": "json_obj", + "description": "The JSON object", + "required": true + } + ] + }, + { + "kind": "unknown", + "name": "dynamic_in", + "concept_ref": "native.Dynamic", + "description": "A dynamic concept", + "required": true, + "presence": "plain", + "gating": true + }, + { + "kind": "unknown", + "name": "composite_in", + "concept_ref": "native.Composite", + "description": "A named composition of contents", + "required": true, + "presence": "plain", + "gating": true + } + ] } } diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.json new file mode 100644 index 0000000..9b885ee --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.json @@ -0,0 +1,16 @@ +{ + "json_in": { + "concept": "native.JSON", + "content": { + "json_obj": {} + } + }, + "dynamic_in": { + "concept": "native.Dynamic", + "content": {} + }, + "composite_in": { + "concept": "native.Composite", + "content": {} + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.toml new file mode 100644 index 0000000..1662455 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.compact.toml @@ -0,0 +1,6 @@ +# concept: native.JSON +json_in = {concept = "native.JSON", content = {json_obj = {}}} +# concept: native.Dynamic +dynamic_in = {concept = "native.Dynamic", content = {}} +# concept: native.Composite +composite_in = {concept = "native.Composite", content = {}} diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.json new file mode 100644 index 0000000..9b885ee --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.json @@ -0,0 +1,16 @@ +{ + "json_in": { + "concept": "native.JSON", + "content": { + "json_obj": {} + } + }, + "dynamic_in": { + "concept": "native.Dynamic", + "content": {} + }, + "composite_in": { + "concept": "native.Composite", + "content": {} + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.toml new file mode 100644 index 0000000..3890494 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_open_natives.explicit.toml @@ -0,0 +1,14 @@ +[json_in] +concept = "native.JSON" + +[json_in.content.json_obj] + +[dynamic_in] +concept = "native.Dynamic" + +[dynamic_in.content] + +[composite_in] +concept = "native.Composite" + +[composite_in.content] diff --git a/tests/fixtures/protocol/inputs_template/manifest.json b/tests/fixtures/protocol/inputs_template/manifest.json index 1d971d6..d91e135 100644 --- a/tests/fixtures/protocol/inputs_template/manifest.json +++ b/tests/fixtures/protocol/inputs_template/manifest.json @@ -11,7 +11,8 @@ "input_semantics_probe.probe_refined", "input_semantics_probe.probe_single", "input_semantics_scaffold.scaffold_fixed_batch", - "input_semantics_scaffold.scaffold_nesting" + "input_semantics_scaffold.scaffold_nesting", + "input_semantics_scaffold.scaffold_open_natives" ], "shapes": [ "compact", @@ -280,6 +281,32 @@ "expected": "url_value" } ] + }, + { + "divergence_id": "unknown-empty-object", + "reason": "An unknown node renders as the empty object, because the descriptor withholds the payload shape at that position and a projection that invented one would stop projecting the descriptor. The engine reflects the runtime content class instead and fills a required dict with a sample key/value pair whoever fills the template in has to delete. The empty object is re-shapable: the round-trip takes it back.", + "ledger_item": null, + "occurrences": 2, + "examples": [ + { + "pipe_ref": "input_semantics_scaffold.scaffold_open_natives", + "shape": "compact", + "path": "json_in.content.json_obj", + "engine": { + "json_obj_key": "json_obj_value" + }, + "expected": {} + }, + { + "pipe_ref": "input_semantics_scaffold.scaffold_open_natives", + "shape": "explicit", + "path": "json_in.content.json_obj", + "engine": { + "json_obj_key": "json_obj_value" + }, + "expected": {} + } + ] } ], "unshapeable": [ diff --git a/tests/fixtures/protocol/pipe_io_contracts.json b/tests/fixtures/protocol/pipe_io_contracts.json index 7be1354..fc82d63 100644 --- a/tests/fixtures/protocol/pipe_io_contracts.json +++ b/tests/fixtures/protocol/pipe_io_contracts.json @@ -3387,5 +3387,62 @@ "item_count": null, "optional": false } + }, + "input_semantics_scaffold.scaffold_open_natives": { + "inputs": { + "json_in": { + "concept_ref": "native.JSON", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "properties": { + "json_obj": { + "additionalProperties": true, + "description": "The JSON object", + "title": "Json Obj", + "type": "object" + } + }, + "required": [ + "json_obj" + ], + "title": "native.JSON", + "type": "object", + "description": "A JSON object" + } + }, + "dynamic_in": { + "concept_ref": "native.Dynamic", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "properties": {}, + "title": "native.Dynamic", + "type": "object", + "description": "A dynamic concept" + } + }, + "composite_in": { + "concept_ref": "native.Composite", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "additionalProperties": true, + "description": "A named composition of contents", + "properties": {}, + "title": "native.Composite", + "type": "object" + } + } + }, + "output": { + "concept_ref": "native.Text", + "multiplicity": "single", + "item_count": null, + "optional": false + } } } diff --git a/tests/unit/test_inputs_template_projection.py b/tests/unit/test_inputs_template_projection.py index e6f54e2..9011b24 100644 --- a/tests/unit/test_inputs_template_projection.py +++ b/tests/unit/test_inputs_template_projection.py @@ -51,6 +51,7 @@ _MANIFEST: dict[str, Any] = json.loads((_TEMPLATES_DIR / _MANIFEST_FILE_NAME).read_text(encoding="utf-8")) _INPUT_FORM: dict[str, Any] = json.loads((_CORPUS_DIR / "input_form.json").read_text(encoding="utf-8")) +_PIPE_IO_CONTRACTS: dict[str, Any] = json.loads((_CORPUS_DIR / "pipe_io_contracts.json").read_text(encoding="utf-8")) # The axes are read from the manifest but not trusted from it. Every case below is a product of # these two lists, so a regeneration that dropped a shape or a format would quietly shrink the suite @@ -65,6 +66,9 @@ _DIVERGENCES: list[dict[str, Any]] = _MANIFEST["divergences"] _UNSHAPEABLE: list[dict[str, Any]] = _MANIFEST["unshapeable"] +# One bullet of the README's divergence list: the id in backticks, then the em dash its prose follows. +_README_DIVERGENCE_BULLET = re.compile(r"^- `([a-z0-9-]+)` — ", re.MULTILINE) + # A workspace-ledger id, the form every entry must name its tracking gap in. _LEDGER_ITEM_PATTERN = re.compile(r"^L-\d{6}-[0-9a-f]{6}$") @@ -139,6 +143,12 @@ def test_it_holds_every_pipe_in_both_shapes_and_both_formats(self): def test_it_describes_exactly_the_pipes_the_descriptor_capture_holds(self): assert set(_PIPE_REFS) == set(_INPUT_FORM) + def test_the_two_payload_files_name_the_same_pipes(self): + # `input_form.json` and `pipe_io_contracts.json` are one capture taken in one command, so a + # pipe present in either must be present in both. Nothing else here reads the contracts file, + # which is what let one half lose a pipe with the whole suite still green. + assert set(_PIPE_IO_CONTRACTS) == set(_INPUT_FORM) + @pytest.mark.parametrize(("pipe_ref", "shape", "file_format"), _CASES) def test_no_template_is_empty(self, pipe_ref: str, shape: str, file_format: str): assert _read_template(pipe_ref=pipe_ref, shape=shape, file_format=file_format).strip() @@ -150,6 +160,17 @@ class TestDeclaredDivergences: def test_at_least_one_is_declared(self): assert _DIVERGENCES + def test_every_declared_class_is_documented_in_the_readme_and_no_other(self): + # The README beside the corpus explains each class in prose, and that is the half of the record + # nothing enforced: the manifest is checked hard, so a class that stops occurring has to be + # retired deliberately, but a retired class could leave its bullet behind describing a difference + # that no longer exists — and the suite, parametrised over the manifest's own list, would simply + # shrink rather than fail. The equality runs both ways, so neither half can move alone. + readme = (_CORPUS_DIR / "README.md").read_text(encoding="utf-8") + documented = set(_README_DIVERGENCE_BULLET.findall(readme)) + declared = {str(divergence["divergence_id"]) for divergence in _DIVERGENCES} + assert documented == declared + @pytest.mark.parametrize("divergence", _DIVERGENCES, ids=lambda divergence: str(divergence["divergence_id"])) def test_a_divergence_states_why_it_exists_and_where(self, divergence: dict[str, Any]): assert divergence["reason"] From ab27956cbe836362926d772050b0ec1ea2ed94c0 Mon Sep 17 00:00:00 2001 From: Thomas Hebrard Date: Wed, 2 Sep 2026 15:55:38 +0200 Subject: [PATCH 02/11] feature: the output-form descriptor, and a payload schema on the output contract (#59) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirrors the standard's new Output-Form Descriptor page and the `json_schema` member it added to the output contract, which landed in one version because neither is useful without the other. `mthds.protocol.output_form` declares no node type: `field` is `InputFormField`, reused verbatim — an output is a concept reference exactly like an input is, so its kinds, its nesting and its constraints are the same questions with the same answers, and a second node union would be a second place for them to drift. One `field`, not a `fields` list: a pipe has exactly one output. What the module adds is the enforcement of what an output does NOT carry — `presence` and `gating` are optional on the node so a slotless node can exist, not so a producer may fill them in with something plausible, and one that does now fails the parse. `PipeOutputContract.json_schema` is breaking: the shape is closed and the member required, so a payload from an earlier engine fails the parse. It is the PAYLOAD's schema, which is where its rule departs from its input twin's — an input's describes what a caller sends, an output's what comes back, so a plural output's is the list content model rather than a bare array. The parity capture gains an output_form.json and a fourth corpus bundle covering outputs specifically. That bundle exists because the three before it all resolved to Text: a corpus whose every pipe outputs one native scalar can only capture one output descriptor and one output schema, so the plural wrap — the producer obligation that fails silently — would never have appeared in it. Claude-Session: https://claude.ai/code/session_019VJ89z3Gza9CW25pQombFK Co-authored-by: Claude Opus 5 (1M context) --- CHANGELOG.md | 14 + mthds/protocol/output_form.py | 132 ++++ mthds/protocol/pipe_io_contracts.py | 52 +- tests/fixtures/protocol/input_form.json | 91 +++ ...cs_output.output_fixed_plural.compact.json | 3 + ...cs_output.output_fixed_plural.compact.toml | 2 + ...s_output.output_fixed_plural.explicit.json | 8 + ...s_output.output_fixed_plural.explicit.toml | 5 + ...cs_output.output_native_image.compact.json | 3 + ...cs_output.output_native_image.compact.toml | 2 + ...s_output.output_native_image.explicit.json | 8 + ...s_output.output_native_image.explicit.toml | 5 + ...ics_output.output_native_text.compact.json | 3 + ...ics_output.output_native_text.compact.toml | 2 + ...cs_output.output_native_text.explicit.json | 8 + ...cs_output.output_native_text.explicit.toml | 5 + ...emantics_output.output_nested.compact.json | 3 + ...emantics_output.output_nested.compact.toml | 2 + ...mantics_output.output_nested.explicit.json | 8 + ...mantics_output.output_nested.explicit.toml | 5 + ...antics_output.output_optional.compact.json | 3 + ...antics_output.output_optional.compact.toml | 2 + ...ntics_output.output_optional.explicit.json | 8 + ...ntics_output.output_optional.explicit.toml | 5 + ...emantics_output.output_plural.compact.json | 3 + ...emantics_output.output_plural.compact.toml | 2 + ...mantics_output.output_plural.explicit.json | 8 + ...mantics_output.output_plural.explicit.toml | 5 + ...tics_output.output_structured.compact.json | 3 + ...tics_output.output_structured.compact.toml | 2 + ...ics_output.output_structured.explicit.json | 8 + ...ics_output.output_structured.explicit.toml | 5 + .../protocol/inputs_template/manifest.json | 10 +- tests/fixtures/protocol/output_form.json | 281 +++++++ .../fixtures/protocol/pipe_io_contracts.json | 714 +++++++++++++++++- tests/unit/test_data.py | 34 +- tests/unit/test_protocol_output_form.py | 143 ++++ tests/unit/test_protocol_pipe_io_contracts.py | 5 +- 38 files changed, 1578 insertions(+), 24 deletions(-) create mode 100644 mthds/protocol/output_form.py create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.toml create mode 100644 tests/fixtures/protocol/output_form.json create mode 100644 tests/unit/test_protocol_output_form.py diff --git a/CHANGELOG.md b/CHANGELOG.md index dd2f717..0631237 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,20 @@ - Updated `README.md` and `docs/runners.md` to reference the new client-side inputs-template projection. +- **`mthds.protocol.output_form` — the output-form descriptor, mirroring the standard's new page.** `PipeOutputFormDescriptor` carries one `field`, not a `fields` list: a pipe has exactly one output where it may have many inputs, which is the only shape difference from its input twin and follows from the language. `OutputForm` is the `pipe_ref` → descriptor map, over the same key set as `pipe_io_contracts` and `input_form`. + + The module declares no node type. `field` is `InputFormField`, reused verbatim — an output is a concept reference exactly like an input is, so its kinds, its nesting and its constraints are the same questions with the same answers, and a second node union would be a second place for them to drift. What it adds is the enforcement of what an output does NOT carry: `presence` and `gating` are facts of an input slot, both already optional on the node so that a slotless node could exist, and a producer that fills them in with something plausible now fails the parse. The wire-null rule (`title: None` is absent, never `null`) is stated on the container for the same reason its input twin states it there — only at that level can a raw wire mapping be told from the keyword arguments of a programmatic construction. + +- **`PipeOutputContract.json_schema` — the payload's schema (breaking).** The output contract was a closed shape with four members and no schema, on the reasoning that a run's payload is the run's own result. That answers *what did this run produce?*, correctly not a contract's business, rather than the question a consumer asks: *what shape will it be?* — which is declared in the source and knowable before any run. + + **It is the PAYLOAD's schema, and that is where its rule departs from the input twin's.** An input's describes what a caller *sends*, so a plural slot's is a bare array. An output's describes what *comes back*, which is the concept's content model — so a `native.Text` output's is the object declaring `text`, and a `Concept[]` output's is the list content model rather than a bare array. On the fixed arm the element array carries `minItems`/`maxItems` equal to `item_count`; since `optional` already states the output may be absent, the schema describes the shape when present and carries no null arm. + + Breaking because the shape is closed and the member is required: a payload from an earlier engine fails the parse, which is the intended behaviour for a minor version of the artifact. + +- **`tests/unit/test_protocol_output_form.py`** — parity against the new `output_form.json` capture, and the rules a type cannot state: the shared key set, the absence of both slot facts, and the **plural wrap** (a plural output described as a `list`, a single one not), which is the one place producing this artifact is real work rather than reuse and the one that fails silently. It also asserts what the corpus must reach — both plural arms — since a capture with no plural output could not have caught the wrap being skipped at all. + +- **The shared projection fixture corpus.** `tests/fixtures/protocol/inputs_template/` pairs each captured pipe's input-form descriptor with the fill-in inputs template a client-side projection must produce from it — the compact and explicit shapes, as JSON and as TOML — and `tests/unit/test_inputs_template_projection.py` holds the corpus to five properties: the whole closed `FieldKind` vocabulary is exercised, every pipe is present in both shapes and both formats, each deliberate departure from the reference engine's own renderer is still visible in the committed bytes, the manifest's `unshapeable` record stays keyed to this corpus, and (skip-gated until `mthds.protocol.inputs_template` lands) the projection reproduces every file byte for byte. That `unshapeable` array is the capture's own statement about which of its pinned templates the runtime's input shaper refuses today: the generator round-trips every projected template through `InputShaper.shape` and records each refusal with the error's class name and the ledger item whose fix retires it, so a template that cannot be filled in and handed back is declared rather than discovered. The corpus is committed identically in `mthds-js`, which runs the twin of that suite, and `conformance` compares the two mirrors file by file — that byte identity is the point, since the two projections are written in different languages and must agree exactly, TOML comment lines included. The descriptor and contract captures are regenerated from the same command and gain a third bundle covering a text field merely named `url` beside a real file position, optional file fields nested inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept, plus a fourth covering OUTPUTS — flat and nested structures, both plural arms, an optional output, a wrapping native scalar and a file-bearing one. That fourth bundle exists because the three before it all resolved to `Text`: a corpus whose every pipe outputs one native scalar can only capture one output descriptor and one output schema, and the two facts most likely to be got wrong would never appear in it. + ## [v0.11.1] - 2026-08-28 ### Fixed diff --git a/mthds/protocol/output_form.py b/mthds/protocol/output_form.py new file mode 100644 index 0000000..da678c9 --- /dev/null +++ b/mthds/protocol/output_form.py @@ -0,0 +1,132 @@ +"""Wire models for the output-form descriptor — mirrors `docs/spec/output-form-descriptor.md`. + + output_form : OutputForm = dict[pipe_ref, PipeOutputFormDescriptor] + PipeOutputFormDescriptor.field : InputFormField + +The presentation view of what a pipe RESOLVES TO, and the exact twin of `input_form` for +the other half of the contract. Where the input-form descriptor states, in authored order, +what each declared input slot is, this states what the single output is: its kind, its +nesting, its constraints — everything a consumer needs to render a result, register a tool +signature with a return type, or project a typed structure for it. + +## An output is a concept ref exactly like an input is + +That is the whole design, and it is why this module declares almost nothing of its own. +The same concepts, the same structures, the same field kinds, the same nesting, so the same +node vocabulary: `field` is an `InputFormField`, reused verbatim from +`mthds.protocol.input_form`. There is no parallel node union, no second `kind` enum, and no +second place for kinds to drift. + +What differs is only the SLOT facts, and there are three: + +- **`name`.** An input's name is authored by the method; an output has none to author. The + node still carries one, because `InputFormField` is the named half of the union and a + nameless node is the shape a `list`'s `item` holds. A producer states `"output"`. Nothing + reads it: a result is labelled by its concept, and a list entry by its index — the same + rule this page's input twin already fixes for list items. It is an address, not a label. +- **`presence`.** Three-valued on an input; an output has no marker to state — `!` MUST NOT + appear on one, and `?` is stated by the contract's `optional`. +- **`gating`.** Whether Run waits for the slot. Meaningless for a result. + +All three of the latter two are already optional on the node, precisely so a node can exist +without them, and this module enforces their ABSENCE the way the input descriptor enforces +their presence. A producer that stamps a slot fact onto an output node fails the parse. + +## One `field`, not a `fields` list + +The one shape difference from `PipeInputFormDescriptor`, and it follows from the language +rather than from taste: a pipe has exactly one output where it may have many inputs. A list +of one would invite a consumer to loop, and a producer to wonder what a second entry means. + +## Plurality is on the DESCRIPTOR, never on the concept + +`concept_ref` is the element with any multiplicity suffix stripped, on both sides of the +contract — a `Concept[]` output names `Concept`. So a plural output is a `list` node whose +`item` is the element node, exactly as a plural input's descriptor is, and a producer must +perform that wrap from the contract's `multiplicity`: the concept alone cannot state it. + +This is the one place producing this artifact is real work rather than reuse, and it fails +silently when skipped — a `Concept[]` output described as its element renders one item where +a run produced many. A consumer never sees the wrap: it reads `kind: "list"` and never +touches the contract for plurality. + +## Read with the contract, never instead of it + +The descriptor states what the field IS; `PipeOutputContract.json_schema` states the shape +of the payload it arrives in, and names the property that payload sits under. Neither is +sufficient alone, which is why the two landed in the same version of the standard. + +Closed shapes, on the same policy as every artifact model here: a member this version does +not define is version drift, rejected at the parse. +""" + +from __future__ import annotations + +from typing import Any, Self, TypeAlias, cast + +from pydantic import BaseModel, ConfigDict, model_validator + +from mthds.protocol.input_form import ( + InputFormField, + _check_no_explicit_nulls, # pyright: ignore[reportPrivateUsage] + _check_pipe_slot_facts_absent, # pyright: ignore[reportPrivateUsage] + _raw_node_label, # pyright: ignore[reportPrivateUsage] +) + +__all__ = ["OutputForm", "PipeOutputFormDescriptor"] + + +class PipeOutputFormDescriptor(BaseModel): + """The output form of one pipe — one `output_form` entry, carrying the single output node. + + Closed shape (`extra="forbid"`). The node itself is an `InputFormField`, so it is parsed by + the same discriminated union every input node is, and a consumer narrows it the same way. + """ + + model_config = ConfigDict(extra="forbid") + + field: InputFormField + """What the pipe resolves to, described. A `list` node on a plural output, whose `item` is the + element; the element concept is named by `concept_ref` on both, suffix stripped.""" + + @model_validator(mode="before") + @classmethod + def reject_explicit_wire_nulls(cls, data: Any) -> Any: + """A wire payload never spells absence as `null` — the same rule `PipeInputFormDescriptor` states, for the same reason. + + Enforced on the container rather than on the node base because only here can a raw wire + mapping be told apart from the keyword arguments of a programmatic construction, where + building a node with `title=None` is the supported idiom. + """ + if not isinstance(data, dict): + return data + raw_form = cast("dict[str, Any]", data) + raw_field = raw_form.get("field") + if isinstance(raw_field, dict): + raw_node = cast("dict[str, Any]", raw_field) + _check_no_explicit_nulls(node=raw_node, position=f"The output {_raw_node_label(node=raw_node, fallback='field')}") + return raw_form + + @model_validator(mode="after") + def validate_no_pipe_slot_facts(self) -> Self: + """An output node states neither `presence` nor `gating`: both are facts of an INPUT slot, and an output has none. + + The mirror image of `PipeInputFormDescriptor.validate_top_level_placement`, which demands + both on a top-level input field. Enforcing the absence rather than ignoring a stray value + is what keeps the reused node type honest: `presence` and `gating` are optional on it so + that a slotless node can exist, not so that a producer may fill them in with something + plausible. + """ + _check_pipe_slot_facts_absent(node=self.field, position=f"The output {self.field.node_label}") + return self + + +OutputForm: TypeAlias = dict[str, PipeOutputFormDescriptor] +"""The `output_form` artifact: namespaced `pipe_ref` (`domain_path.pipe_code`) → the pipe's output form. + +The same key set as `pipe_io_contracts` and `input_form` — every pipe in the resolved library has +an entry, contract-only pipe signatures included. The artifact arrives as an extension field of the +validate report, so a consumer parses it by declaring a typed field — `output_form: OutputForm | +None = None` on a model extending the report — and pydantic parses the map and the discriminated +union from that plain annotation; no adapter machinery is involved. +""" diff --git a/mthds/protocol/pipe_io_contracts.py b/mthds/protocol/pipe_io_contracts.py index ff798f1..ffabb32 100644 --- a/mthds/protocol/pipe_io_contracts.py +++ b/mthds/protocol/pipe_io_contracts.py @@ -7,8 +7,15 @@ A pipe I/O contract states, for one pipe, exactly what a caller must supply and what the pipe resolves to: for each declared input, the concept it expects, its authored presence marker, how many items it takes and the JSON Schema its content must satisfy; for the -output, the concept it produces and how many items that is. It is a projection of the -resolved library, not a second declaration of it. +output, the concept it produces, how many items that is, whether it may be absent, and the +JSON Schema of the payload it resolves to. It is a projection of the resolved library, not +a second declaration of it. + +The two schemas answer different questions, and conflating them is the one mistake this +artifact invites. An input's `json_schema` describes what a caller **sends**, so a plural +slot's is a bare array. An output's describes what **comes back**, which is the concept's +content model, so a plural output's is that model's list envelope rather than a bare array. +Both are declared facts, knowable before any run happens; neither is the run's result. The artifact is a **recommended extension field** of the `POST /validate` valid report, where it rides the field name `pipe_io_contracts` (`ValidationReport.model_extra`). It is @@ -150,13 +157,22 @@ def validate_presence_pairing(self) -> Self: class PipeOutputContract(BaseModel): - """What the pipe resolves to: the concept it produces, how many items that is, and whether it may be absent. - - Deliberately asymmetric with the input side: an output carries a two-valued `optional` where - an input carries a three-valued `presence`, because `!` MUST NOT appear on an output — a force - marker is a use-site assertion about an input, so a three-valued output slot would have an arm - nothing can ever produce. No output member carries a schema: the payload a run produces is the - run's own result. Closed shape (`extra="forbid"`). + """What the pipe resolves to: the concept it produces, how many items that is, whether it may be absent, and the shape of its payload. + + Deliberately asymmetric with the input side in exactly one place, and no longer in a second. + An output carries a two-valued `optional` where an input carries a three-valued `presence`, + because `!` MUST NOT appear on an output — a force marker is a use-site assertion about an + input, so a three-valued output slot would have an arm nothing can ever produce. That + asymmetry is a language fact and it stays. + + The schema asymmetry was not. An output used to carry none, on the reasoning that "the payload + a run produces is the run's own result" — which answers *what did this run produce?*, correctly + not a contract's business, rather than the question a consumer actually asks: *what shape will + it be?* That is declared in the `.mthds` source and knowable before any run happens, exactly as + it is on the input side. Withholding it left every consumer inferring a payload's shape by + inspecting it, which is guessing at the standard rather than reading it. + + Closed shape (`extra="forbid"`). """ model_config = ConfigDict(extra="forbid") @@ -174,6 +190,24 @@ class PipeOutputContract(BaseModel): a recorded absence instead of a value — not that the run may fail. Never `True` on a plural output: `?` may not be combined with multiplicity, and an absent plural is the empty list.""" + json_schema: dict[str, Any] + """The JSON Schema of the payload the pipe resolves to — the concept's CONTENT MODEL, not a + caller's argument, which is where this member's rule departs from its input twin. + + A `native.Text` output resolves to that concept's content model, so its schema is the object + declaring `text`; a `Concept[]` output resolves to the list content model, so its schema is the + object declaring the element array. Stating a bare array here — the input side's plural rule — + would describe a payload no runtime produces. + + Two rules and nothing else. On the fixed arm the element array carries `minItems` and `maxItems` + equal to `item_count`. And `optional: true` already states that the output may be absent, so the + schema describes the shape **when present** — never a null arm. + + Read together with the output-form descriptor (`mthds.protocol.output_form`), which states the + node's `kind`: neither is sufficient alone. The descriptor says what the field IS, the schema + names the property its payload sits under, and a renderer that has one but not the other is back + to inferring the other from the value.""" + @model_validator(mode="after") def validate_item_count_pairing(self) -> Self: _check_item_count_pairing(multiplicity=self.multiplicity, item_count=self.item_count) diff --git a/tests/fixtures/protocol/input_form.json b/tests/fixtures/protocol/input_form.json index 6f8fb5f..5919277 100644 --- a/tests/fixtures/protocol/input_form.json +++ b/tests/fixtures/protocol/input_form.json @@ -1483,5 +1483,96 @@ "gating": true } ] + }, + "input_semantics_output.output_structured": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_nested": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_plural": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_fixed_plural": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_optional": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_native_text": { + "fields": [ + { + "kind": "prose", + "name": "note", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_output.output_native_image": { + "fields": [ + { + "kind": "prose", + "name": "subject", + "concept_ref": "native.Text", + "description": "A text", + "required": true, + "presence": "plain", + "gating": true + } + ] } } diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_fixed_plural.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.json new file mode 100644 index 0000000..c4dbcc5 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.json @@ -0,0 +1,3 @@ +{ + "subject": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.toml new file mode 100644 index 0000000..d07e1c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +subject = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.json new file mode 100644 index 0000000..7385ef0 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.json @@ -0,0 +1,8 @@ +{ + "subject": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.toml new file mode 100644 index 0000000..695d76a --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_image.explicit.toml @@ -0,0 +1,5 @@ +[subject] +concept = "native.Text" + +[subject.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_native_text.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_nested.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_optional.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_plural.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.json new file mode 100644 index 0000000..e4942fc --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.json @@ -0,0 +1,3 @@ +{ + "note": "text_value" +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.toml new file mode 100644 index 0000000..94700c6 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Text +note = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.json new file mode 100644 index 0000000..853cf7d --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.json @@ -0,0 +1,8 @@ +{ + "note": { + "concept": "native.Text", + "content": { + "text": "text_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.toml new file mode 100644 index 0000000..6c24aac --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_output.output_structured.explicit.toml @@ -0,0 +1,5 @@ +[note] +concept = "native.Text" + +[note.content] +text = "text_value" diff --git a/tests/fixtures/protocol/inputs_template/manifest.json b/tests/fixtures/protocol/inputs_template/manifest.json index d91e135..c8ee5d6 100644 --- a/tests/fixtures/protocol/inputs_template/manifest.json +++ b/tests/fixtures/protocol/inputs_template/manifest.json @@ -2,10 +2,18 @@ "bundles": [ "hinted_bundle.mthds", "probe_bundle.mthds", - "scaffold_bundle.mthds" + "scaffold_bundle.mthds", + "output_bundle.mthds" ], "pipes": [ "input_semantics_hinted.hinted_slots", + "input_semantics_output.output_fixed_plural", + "input_semantics_output.output_native_image", + "input_semantics_output.output_native_text", + "input_semantics_output.output_nested", + "input_semantics_output.output_optional", + "input_semantics_output.output_plural", + "input_semantics_output.output_structured", "input_semantics_probe.probe_markers", "input_semantics_probe.probe_native_inputs", "input_semantics_probe.probe_refined", diff --git a/tests/fixtures/protocol/output_form.json b/tests/fixtures/protocol/output_form.json new file mode 100644 index 0000000..a460752 --- /dev/null +++ b/tests/fixtures/protocol/output_form.json @@ -0,0 +1,281 @@ +{ + "input_semantics_hinted.hinted_slots": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_probe.probe_single": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_probe.probe_native_inputs": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_probe.probe_markers": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_probe.probe_refined": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_scaffold.scaffold_nesting": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_scaffold.scaffold_fixed_batch": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_scaffold.scaffold_open_natives": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_output.output_structured": { + "field": { + "kind": "object", + "name": "output", + "concept_ref": "input_semantics_output.Verdict", + "description": "OUTPUT_desc_concept_Verdict: a flat structured result, so an output descriptor recurses at least once", + "required": true, + "fields": [ + { + "kind": "enum", + "name": "label", + "description": "OUTPUT_desc_field_label: the verdict itself", + "required": true, + "choices": [ + "accepted", + "rejected", + "deferred" + ] + }, + { + "kind": "number", + "name": "confidence", + "description": "OUTPUT_desc_field_confidence: how sure, 0 to 1", + "required": true, + "integer": false + }, + { + "kind": "text", + "name": "rationale", + "description": "OUTPUT_desc_field_rationale: why it landed there", + "required": false + } + ] + } + }, + "input_semantics_output.output_nested": { + "field": { + "kind": "object", + "name": "output", + "concept_ref": "input_semantics_output.Dossier", + "description": "OUTPUT_desc_concept_Dossier: a nested structured result carrying a date and a list of concepts", + "required": true, + "fields": [ + { + "kind": "text", + "name": "reference", + "description": "OUTPUT_desc_field_reference: the dossier reference", + "required": true + }, + { + "kind": "date", + "name": "opened_on", + "description": "OUTPUT_desc_field_opened_on: when it was opened", + "required": true, + "datetime": false + }, + { + "kind": "boolean", + "name": "settled", + "description": "OUTPUT_desc_field_settled: whether it is closed", + "required": false + }, + { + "kind": "list", + "name": "findings", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_field_findings: a NESTED list, which stays a bare array on the wire", + "required": true, + "item": { + "kind": "object", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "required": true, + "fields": [ + { + "kind": "text", + "name": "summary", + "description": "OUTPUT_desc_field_summary: what was found", + "required": true + }, + { + "kind": "number", + "name": "severity", + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "required": true, + "integer": true + } + ] + } + } + ] + } + }, + "input_semantics_output.output_plural": { + "field": { + "kind": "list", + "name": "output", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "required": true, + "item": { + "kind": "object", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "required": true, + "fields": [ + { + "kind": "text", + "name": "summary", + "description": "OUTPUT_desc_field_summary: what was found", + "required": true + }, + { + "kind": "number", + "name": "severity", + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "required": true, + "integer": true + } + ] + } + } + }, + "input_semantics_output.output_fixed_plural": { + "field": { + "kind": "list", + "name": "output", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "required": true, + "item": { + "kind": "object", + "concept_ref": "input_semantics_output.Finding", + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "required": true, + "fields": [ + { + "kind": "text", + "name": "summary", + "description": "OUTPUT_desc_field_summary: what was found", + "required": true + }, + { + "kind": "number", + "name": "severity", + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "required": true, + "integer": true + } + ] + }, + "item_count": 3 + } + }, + "input_semantics_output.output_optional": { + "field": { + "kind": "object", + "name": "output", + "concept_ref": "input_semantics_output.Verdict", + "description": "OUTPUT_desc_concept_Verdict: a flat structured result, so an output descriptor recurses at least once", + "required": true, + "fields": [ + { + "kind": "enum", + "name": "label", + "description": "OUTPUT_desc_field_label: the verdict itself", + "required": true, + "choices": [ + "accepted", + "rejected", + "deferred" + ] + }, + { + "kind": "number", + "name": "confidence", + "description": "OUTPUT_desc_field_confidence: how sure, 0 to 1", + "required": true, + "integer": false + }, + { + "kind": "text", + "name": "rationale", + "description": "OUTPUT_desc_field_rationale: why it landed there", + "required": false + } + ] + } + }, + "input_semantics_output.output_native_text": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_output.output_native_image": { + "field": { + "kind": "image", + "name": "output", + "concept_ref": "native.Image", + "description": "An image", + "required": true + } + } +} diff --git a/tests/fixtures/protocol/pipe_io_contracts.json b/tests/fixtures/protocol/pipe_io_contracts.json index fc82d63..2ad66d0 100644 --- a/tests/fixtures/protocol/pipe_io_contracts.json +++ b/tests/fixtures/protocol/pipe_io_contracts.json @@ -145,7 +145,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_probe.probe_single": { @@ -736,7 +751,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_probe.probe_native_inputs": { @@ -1338,7 +1368,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_probe.probe_markers": { @@ -2447,7 +2492,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_probe.probe_refined": { @@ -2627,7 +2687,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_scaffold.scaffold_nesting": { @@ -3002,7 +3077,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_scaffold.scaffold_fixed_batch": { @@ -3385,7 +3475,22 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } } }, "input_semantics_scaffold.scaffold_open_natives": { @@ -3442,7 +3547,600 @@ "concept_ref": "native.Text", "multiplicity": "single", "item_count": null, - "optional": false + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "input_semantics_output.output_structured": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "input_semantics_output.Verdict", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "description": "OUTPUT_desc_concept_Verdict: a flat structured result, so an output descriptor recurses at least once", + "properties": { + "label": { + "description": "OUTPUT_desc_field_label: the verdict itself", + "enum": [ + "accepted", + "rejected", + "deferred" + ], + "title": "Label", + "type": "string" + }, + "confidence": { + "description": "OUTPUT_desc_field_confidence: how sure, 0 to 1", + "title": "Confidence", + "type": "number" + }, + "rationale": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "OUTPUT_desc_field_rationale: why it landed there", + "title": "Rationale" + } + }, + "required": [ + "label", + "confidence" + ], + "title": "input_semantics_output.Verdict", + "type": "object" + } + } + }, + "input_semantics_output.output_nested": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "input_semantics_output.Dossier", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "$defs": { + "input_semantics_output__Finding": { + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "properties": { + "summary": { + "description": "OUTPUT_desc_field_summary: what was found", + "title": "Summary", + "type": "string" + }, + "severity": { + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "title": "Severity", + "type": "integer" + } + }, + "required": [ + "summary", + "severity" + ], + "title": "input_semantics_output__Finding", + "type": "object" + } + }, + "description": "OUTPUT_desc_concept_Dossier: a nested structured result carrying a date and a list of concepts", + "properties": { + "reference": { + "description": "OUTPUT_desc_field_reference: the dossier reference", + "title": "Reference", + "type": "string" + }, + "opened_on": { + "description": "OUTPUT_desc_field_opened_on: when it was opened", + "format": "date", + "title": "Opened On", + "type": "string" + }, + "settled": { + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "default": null, + "description": "OUTPUT_desc_field_settled: whether it is closed", + "title": "Settled" + }, + "findings": { + "description": "OUTPUT_desc_field_findings: a NESTED list, which stays a bare array on the wire", + "items": { + "$ref": "#/$defs/input_semantics_output__Finding" + }, + "title": "Findings", + "type": "array" + } + }, + "required": [ + "reference", + "opened_on", + "findings" + ], + "title": "input_semantics_output.Dossier", + "type": "object" + } + } + }, + "input_semantics_output.output_plural": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "input_semantics_output.Finding", + "multiplicity": "variable", + "item_count": null, + "optional": false, + "json_schema": { + "$defs": { + "input_semantics_output__Finding": { + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "properties": { + "summary": { + "description": "OUTPUT_desc_field_summary: what was found", + "title": "Summary", + "type": "string" + }, + "severity": { + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "title": "Severity", + "type": "integer" + } + }, + "required": [ + "summary", + "severity" + ], + "title": "input_semantics_output__Finding", + "type": "object" + } + }, + "properties": { + "items": { + "items": { + "$ref": "#/$defs/input_semantics_output__Finding" + }, + "title": "Items", + "type": "array" + } + }, + "required": [ + "items" + ], + "title": "ListContent[input_semantics_output__Finding]", + "type": "object" + } + } + }, + "input_semantics_output.output_fixed_plural": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "input_semantics_output.Finding", + "multiplicity": "fixed", + "item_count": 3, + "optional": false, + "json_schema": { + "$defs": { + "input_semantics_output__Finding": { + "description": "OUTPUT_desc_concept_Finding: one element of a plural result", + "properties": { + "summary": { + "description": "OUTPUT_desc_field_summary: what was found", + "title": "Summary", + "type": "string" + }, + "severity": { + "description": "OUTPUT_desc_field_severity: how bad, 1 to 5", + "title": "Severity", + "type": "integer" + } + }, + "required": [ + "summary", + "severity" + ], + "title": "input_semantics_output__Finding", + "type": "object" + } + }, + "properties": { + "items": { + "items": { + "$ref": "#/$defs/input_semantics_output__Finding" + }, + "title": "Items", + "type": "array", + "minItems": 3, + "maxItems": 3 + } + }, + "required": [ + "items" + ], + "title": "ListContent[input_semantics_output__Finding]", + "type": "object" + } + } + }, + "input_semantics_output.output_optional": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "input_semantics_output.Verdict", + "multiplicity": "single", + "item_count": null, + "optional": true, + "json_schema": { + "description": "OUTPUT_desc_concept_Verdict: a flat structured result, so an output descriptor recurses at least once", + "properties": { + "label": { + "description": "OUTPUT_desc_field_label: the verdict itself", + "enum": [ + "accepted", + "rejected", + "deferred" + ], + "title": "Label", + "type": "string" + }, + "confidence": { + "description": "OUTPUT_desc_field_confidence: how sure, 0 to 1", + "title": "Confidence", + "type": "number" + }, + "rationale": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "OUTPUT_desc_field_rationale: why it landed there", + "title": "Rationale" + } + }, + "required": [ + "label", + "confidence" + ], + "title": "input_semantics_output.Verdict", + "type": "object" + } + } + }, + "input_semantics_output.output_native_text": { + "inputs": { + "note": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "native.Text", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "input_semantics_output.output_native_image": { + "inputs": { + "subject": { + "concept_ref": "native.Text", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "output": { + "concept_ref": "native.Image", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "description": "An image", + "properties": { + "url": { + "description": "The image URL: a storage URI, an HTTP(S) URL, or a base64 data URL", + "title": "Url", + "type": "string" + }, + "public_url": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The public URL of the image", + "title": "Public Url" + }, + "source_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The source prompt of the image", + "title": "Source Prompt" + }, + "source_negative_prompt": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The source negative prompt of the image", + "title": "Source Negative Prompt" + }, + "caption": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The caption of the image", + "title": "Caption" + }, + "mime_type": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The MIME type of the image", + "title": "Mime Type" + }, + "width": { + "anyOf": [ + { + "exclusiveMinimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The width of the image, in pixels", + "title": "Width" + }, + "height": { + "anyOf": [ + { + "exclusiveMinimum": 0, + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The height of the image, in pixels", + "title": "Height" + }, + "filename": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The original filename of the image", + "title": "Filename" + } + }, + "required": [ + "url" + ], + "title": "native.Image", + "type": "object" + } } } } diff --git a/tests/unit/test_data.py b/tests/unit/test_data.py index 25b22e7..158c264 100644 --- a/tests/unit/test_data.py +++ b/tests/unit/test_data.py @@ -271,7 +271,13 @@ class PipeIOContractWireNodes: """Hand-written contract entries probing the closed shapes of `mthds.protocol.pipe_io_contracts`.""" _TEXT_SCHEMA: ClassVar[dict[str, Any]] = {"type": "object", "properties": {"text": {"type": "string"}}, "required": ["text"]} - _OUTPUT: ClassVar[dict[str, Any]] = {"concept_ref": "legal.Summary", "multiplicity": "single", "item_count": None, "optional": False} + _OUTPUT: ClassVar[dict[str, Any]] = { + "concept_ref": "legal.Summary", + "multiplicity": "single", + "item_count": None, + "optional": False, + "json_schema": _TEXT_SCHEMA, + } # Rejected at the parse. INPUT_UNKNOWN_MEMBER: ClassVar[dict[str, Any]] = { @@ -330,15 +336,35 @@ class PipeIOContractWireNodes: "item_count": 3, "json_schema": {"type": "array", "items": _TEXT_SCHEMA, "minItems": 3, "maxItems": 3}, } - OUTPUT_UNKNOWN_MEMBER: ClassVar[dict[str, Any]] = {**_OUTPUT, "json_schema": _TEXT_SCHEMA} + OUTPUT_UNKNOWN_MEMBER: ClassVar[dict[str, Any]] = {**_OUTPUT, "presence": "plain"} + """`presence` on an output. The marker is an INPUT-slot fact — `!` may not appear on an output + and `?` is what `optional` states — so this is the member the output shape most plausibly grows + by mistake, which is why it is the one the closed-shape case uses.""" + + OUTPUT_WITHOUT_SCHEMA: ClassVar[dict[str, Any]] = {k: v for k, v in _OUTPUT.items() if k != "json_schema"} + """An output stating no payload schema. Omitted members fail the parse like unknown ones do: + the schema is required precisely so a consumer never has to infer a payload's shape from the + payload.""" OUTPUT_FIXED_WITHOUT_COUNT: ClassVar[dict[str, Any]] = {**_OUTPUT, "multiplicity": "fixed"} - OUTPUT_FIXED_OPTIONAL: ClassVar[dict[str, Any]] = {"concept_ref": "legal.Clause", "multiplicity": "fixed", "item_count": 3, "optional": True} + OUTPUT_FIXED_OPTIONAL: ClassVar[dict[str, Any]] = { + "concept_ref": "legal.Clause", + "multiplicity": "fixed", + "item_count": 3, + "optional": True, + "json_schema": _TEXT_SCHEMA, + } ENTRY_WITHOUT_INPUTS: ClassVar[dict[str, Any]] = {"output": _OUTPUT} ENTRY_UNKNOWN_MEMBER: ClassVar[dict[str, Any]] = {"inputs": {}, "output": _OUTPUT, "description": "Summarize a contract"} # Accepted. ENTRY_WITHOUT_DECLARED_INPUTS: ClassVar[dict[str, Any]] = {"inputs": {}, "output": _OUTPUT} - OUTPUT_SINGLE_OPTIONAL: ClassVar[dict[str, Any]] = {"concept_ref": "legal.Clause", "multiplicity": "single", "item_count": None, "optional": True} + OUTPUT_SINGLE_OPTIONAL: ClassVar[dict[str, Any]] = { + "concept_ref": "legal.Clause", + "multiplicity": "single", + "item_count": None, + "optional": True, + "json_schema": _TEXT_SCHEMA, + } class TomlEmitterCases: diff --git a/tests/unit/test_protocol_output_form.py b/tests/unit/test_protocol_output_form.py new file mode 100644 index 0000000..27d9ad0 --- /dev/null +++ b/tests/unit/test_protocol_output_form.py @@ -0,0 +1,143 @@ +"""Parity and strictness tests for `mthds.protocol.output_form` against the engine-produced fixture. + +The fixture `tests/fixtures/protocol/output_form.json` is the reference engine's own emission, +committed byte-for-byte here and in `mthds-js` alongside its two siblings (the directory's README +carries the provenance). It is parsed the way the artifact actually arrives: as a typed field +declared on a model narrowing the validate report. + +What is worth testing here is narrower than the input side, and deliberately so. The node union is +the SAME one — an output is a concept ref exactly like an input is, so its kinds, its nesting and +its constraints are already covered by `test_protocol_input_form.py`, and re-asserting them here +would be asserting the reuse rather than the artifact. What is new is only what this artifact adds: +the single `field`, the ABSENCE of the two pipe-slot facts, and the plural wrap — the one place +producing this artifact is real work rather than reuse, and the one that fails silently. +""" + +import json +from pathlib import Path +from typing import Any + +import pytest +from pydantic import BaseModel, ConfigDict, ValidationError + +from mthds.protocol.output_form import OutputForm, PipeOutputFormDescriptor +from mthds.protocol.pipe_io_contracts import IOMultiplicity, PipeIOContracts + + +class NarrowedValidateReport(BaseModel): + """How the artifact arrives: a typed field on a model narrowing the validate report.""" + + model_config = ConfigDict(extra="forbid") + + output_form: OutputForm + pipe_io_contracts: PipeIOContracts + + +_FIXTURES = Path(__file__).resolve().parent.parent / "fixtures" / "protocol" +_RAW_OUTPUT_FORM: dict[str, Any] = json.loads((_FIXTURES / "output_form.json").read_text(encoding="utf-8")) +_RAW_CONTRACTS: dict[str, Any] = json.loads((_FIXTURES / "pipe_io_contracts.json").read_text(encoding="utf-8")) +_REPORT = NarrowedValidateReport.model_validate({"output_form": _RAW_OUTPUT_FORM, "pipe_io_contracts": _RAW_CONTRACTS}) +_OUTPUT_FORM = _REPORT.output_form +_CONTRACTS = _REPORT.pipe_io_contracts + + +class TestOutputFormProtocolModels: + @pytest.mark.parametrize("pipe_ref", [pytest.param(pipe_ref, id=pipe_ref) for pipe_ref in _RAW_OUTPUT_FORM]) + def test_fixture_entry_parses_strictly_and_round_trips(self, pipe_ref: str) -> None: + """Each per-pipe descriptor parses under the closed shapes and dumps back to the exact input. + + `exclude_none` is the dump mode, matching the input form's: a slot that does not apply to a + node is ABSENT on this wire, never `null`. + """ + descriptor = PipeOutputFormDescriptor.model_validate(_RAW_OUTPUT_FORM[pipe_ref]) + assert descriptor.model_dump(mode="json", exclude_none=True) == _RAW_OUTPUT_FORM[pipe_ref] + + def test_shares_one_key_set_with_the_contracts(self) -> None: + """All three validate artifacts are keyed by the same `pipe_ref` set — they iterate one pipe sequence.""" + assert sorted(_OUTPUT_FORM) == sorted(_CONTRACTS) + + @pytest.mark.parametrize("pipe_ref", [pytest.param(pipe_ref, id=pipe_ref) for pipe_ref in _RAW_OUTPUT_FORM]) + def test_output_node_states_no_pipe_slot_facts(self, pipe_ref: str) -> None: + """`presence` and `gating` are facts of an INPUT slot, and an output has none. + + Both are optional on the node so a slotless node can exist at all — not so a producer may + fill them in with something plausible. + """ + node = _OUTPUT_FORM[pipe_ref].field + assert node.presence is None + assert node.gating is None + + @pytest.mark.parametrize("pipe_ref", [pytest.param(pipe_ref, id=pipe_ref) for pipe_ref in _RAW_OUTPUT_FORM]) + def test_plurality_is_carried_by_the_descriptor(self, pipe_ref: str) -> None: + """THE producer obligation, and the one that fails silently. + + `concept_ref` is the element with the multiplicity suffix stripped on both sides of the + contract, so a producer that does not read `multiplicity` describes ONE item where a run + returns many — and every renderer then shows one. A consumer never re-derives this: it + reads `kind` and never touches the contract for plurality. + """ + node = _OUTPUT_FORM[pipe_ref].field + if _CONTRACTS[pipe_ref].output.multiplicity.is_plural: + assert node.kind == "list" + else: + assert node.kind != "list" + + def test_the_corpus_reaches_both_plural_arms(self) -> None: + """A corpus with no plural output could not have caught the wrap being skipped at all.""" + multiplicities = {contract.output.multiplicity for contract in _CONTRACTS.values()} + assert IOMultiplicity.VARIABLE in multiplicities + assert IOMultiplicity.FIXED in multiplicities + + def test_rejects_an_output_node_carrying_a_slot_fact(self) -> None: + """Stating the absence is a parse rule, not a convention a producer may quietly break.""" + node = {"name": "output", "kind": "prose", "concept_ref": "native.Text", "required": True} + with pytest.raises(ValidationError): + PipeOutputFormDescriptor.model_validate({"field": {**node, "presence": "plain"}}) + with pytest.raises(ValidationError): + PipeOutputFormDescriptor.model_validate({"field": {**node, "gating": True}}) + + def test_rejects_an_unknown_member_and_an_input_form_shape(self) -> None: + """Closed shape, and `fields` is the input form's shape: a pipe has exactly one output.""" + node = {"name": "output", "kind": "prose", "concept_ref": "native.Text", "required": True} + with pytest.raises(ValidationError): + PipeOutputFormDescriptor.model_validate({"field": node, "fields": []}) + with pytest.raises(ValidationError): + PipeOutputFormDescriptor.model_validate({"fields": [node]}) + + def test_rejects_an_explicit_wire_null(self) -> None: + """A wire payload never spells absence as `null` — the same rule the input descriptor states.""" + with pytest.raises(ValidationError): + PipeOutputFormDescriptor.model_validate( + {"field": {"name": "output", "kind": "prose", "concept_ref": "native.Text", "required": True, "title": None}} + ) + + +class TestOutputPayloadSchema: + """The contract's half of the pair — tested here because it is only useful beside the descriptor.""" + + @pytest.mark.parametrize("pipe_ref", [pytest.param(pipe_ref, id=pipe_ref) for pipe_ref in _RAW_CONTRACTS]) + def test_payload_schema_is_a_content_model_never_a_bare_array(self, pipe_ref: str) -> None: + """Where the output side departs from the input side, asserted rather than only documented. + + An input's schema describes what a caller SENDS, so a plural slot's is a bare array. An + output's describes what COMES BACK, which is a content model — an object — whatever the + multiplicity. A bare array here would state a shape no runtime produces. + """ + schema = _CONTRACTS[pipe_ref].output.json_schema + assert schema.get("type") == "object" + assert isinstance(schema.get("properties"), dict) + + @pytest.mark.parametrize("pipe_ref", [pytest.param(pipe_ref, id=pipe_ref) for pipe_ref in _RAW_CONTRACTS]) + def test_fixed_output_bounds_its_element_array(self, pipe_ref: str) -> None: + """On the fixed arm the element array carries `minItems`/`maxItems` equal to `item_count`. + + Without them a `Concept[N]` output would be indistinguishable from a `Concept[]` one by + schema alone — the list content model states no bounds of its own. + """ + output = _CONTRACTS[pipe_ref].output + if output.multiplicity is not IOMultiplicity.FIXED: + pytest.skip("not the fixed arm") + properties: dict[str, Any] = output.json_schema["properties"] + elements = next(iter(properties.values())) + assert elements["minItems"] == output.item_count + assert elements["maxItems"] == output.item_count diff --git a/tests/unit/test_protocol_pipe_io_contracts.py b/tests/unit/test_protocol_pipe_io_contracts.py index 166818b..d7ca3e4 100644 --- a/tests/unit/test_protocol_pipe_io_contracts.py +++ b/tests/unit/test_protocol_pipe_io_contracts.py @@ -138,13 +138,14 @@ def test_input_contract_is_a_closed_shape(self, node: dict[str, Any]) -> None: @pytest.mark.parametrize( "node", [ - pytest.param(PipeIOContractWireNodes.OUTPUT_UNKNOWN_MEMBER, id="schema on an output"), + pytest.param(PipeIOContractWireNodes.OUTPUT_UNKNOWN_MEMBER, id="presence on an output"), + pytest.param(PipeIOContractWireNodes.OUTPUT_WITHOUT_SCHEMA, id="output without a payload schema"), pytest.param(PipeIOContractWireNodes.OUTPUT_FIXED_WITHOUT_COUNT, id="fixed without a count"), pytest.param(PipeIOContractWireNodes.OUTPUT_FIXED_OPTIONAL, id="fixed marked optional"), ], ) def test_output_contract_is_a_closed_shape(self, node: dict[str, Any]) -> None: - """An output carries no schema, obeys the same item-count pair rule as an input, and is plural or optional but never both.""" + """An output states its payload schema, carries no input-slot member, and is plural or optional but never both.""" with pytest.raises(ValidationError): PipeOutputContract.model_validate(node) From bb534a07e91a4b51dcf16134f517dbb02db70f20 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Wed, 2 Sep 2026 16:14:39 +0200 Subject: [PATCH 03/11] feature: gate the shared protocol fixture corpus in this repo's CI (#60) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `tests/fixtures/protocol/` is committed byte-identically here and in `mthds-js`, and until now nothing in either repo checked that. The only comparison lived in `conformance`, where it runs on a `conformance` pull request rather than on the one that moves the corpus — so a recapture landed here without its `mthds-js` twin was caught weeks later by an unrelated change, and the pairing held only because prose in two pull request bodies asked for it. `.github/workflows/protocol-corpus-parity.yml` checks `mthds-js` out on every pull request and compares the two trees, which fails the pull request that would land a one-sided recapture instead of reporting it afterwards. Two properties make it deployable. It needs no secret: both mirrors are public, so the sibling checkout is unauthenticated and the job runs on a pull request from a fork exactly as it does on one from a branch — a gate that needed `conformance` itself, which is internal, could not have done that. And it resolves the sibling ref as this pull request's own branch name when the sibling carries a branch of that name, falling back to `dev`: a recapture lands as a paired pull request in both mirrors at once, and comparing each against the other's `dev` would leave both red until the first merged. The file is a vendored copy of `conformance/tests/mthds/fixtures/protocol_corpus_parity_workflow.yml` and is byte-identical in both mirrors, which is why it derives which repo it is running in from `github.repository` rather than naming one. `make check-fixture-drift` in `conformance` fails when a copy drifts or goes missing, so the coupling cannot be quietly undone. Advances L-260902-61be99 Claude-Session: https://claude.ai/code/session_012E1hEFhVP2Ze394Aic1v58 Co-authored-by: Claude Opus 5 --- .github/workflows/protocol-corpus-parity.yml | 103 +++++++++++++++++++ CHANGELOG.md | 6 ++ 2 files changed, 109 insertions(+) create mode 100644 .github/workflows/protocol-corpus-parity.yml diff --git a/.github/workflows/protocol-corpus-parity.yml b/.github/workflows/protocol-corpus-parity.yml new file mode 100644 index 0000000..76e4fd2 --- /dev/null +++ b/.github/workflows/protocol-corpus-parity.yml @@ -0,0 +1,103 @@ +# The mirror half of the protocol fixture corpus gate. +# +# `tests/fixtures/protocol/` is committed byte-identically in `mthds-js` and `mthds-python`: the +# reference engine's input-form descriptors and pipe I/O contracts, and beside them the inputs +# templates each repo's own client-side projection must produce from those descriptors. Nothing +# inside one mirror can see the other, so this job checks the sibling out and compares the two +# trees — which is what fails the pull request that would land a one-sided recapture, instead of +# leaving the drift for an unrelated `conformance` change to discover weeks later. +# +# THIS FILE IS A VENDORED COPY. The canonical one is +# `conformance/tests/mthds/fixtures/protocol_corpus_parity_workflow.yml`, and `make +# check-fixture-drift` there fails when a mirror's copy drifts from it. Edit the canonical copy +# and re-vendor both mirrors; do not edit this file in place. It is byte-identical in the two +# mirrors, which is why it derives which repo it is running in rather than naming one. +# +# It needs no secret: both mirrors are public, so the sibling checkout is unauthenticated and the +# job runs on a pull request from a fork exactly as it does on one from a branch. +# +# The comparison's authority is `conformance/scripts/check-protocol-fixture-parity.py` — the same +# tree, the same exclusions. That script runs over two checkouts in a workspace; the `diff` below +# is its shape in a single-repo CI job. The two exclusion sets are restated rather than shared, +# and a disagreement is caught rather than tolerated: an exclusion here the script does not have +# leaves the script red, and one the script has but this file does not leaves this job red. + +name: Protocol corpus parity + +on: + pull_request: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref }} + cancel-in-progress: true + +jobs: + parity: + name: Protocol corpus parity + runs-on: ubuntu-latest + steps: + - name: Resolve this mirror and its sibling + id: mirror + env: + REPOSITORY: ${{ github.repository }} + run: | + set -euo pipefail + case "${REPOSITORY##*/}" in + mthds-js) sibling="${REPOSITORY%/*}/mthds-python" ;; + mthds-python) sibling="${REPOSITORY%/*}/mthds-js" ;; + *) + echo "::error::${REPOSITORY} is not one of the two protocol corpus mirrors; this workflow is vendored into mthds-js and mthds-python only." + exit 1 + ;; + esac + echo "sibling=${sibling}" >> "$GITHUB_OUTPUT" + + # A recapture lands as a pull request in each mirror at once, and each of the pair is red + # against the other's `dev` until the first one merges. The pair shares a branch name, so + # prefer the sibling's branch of that name and fall back to `dev` — which is what every + # pull request that does not touch the corpus gets, and what the second of the pair gets + # once the first has merged. + - name: Resolve the sibling ref + id: sibling + env: + SIBLING: ${{ steps.mirror.outputs.sibling }} + HEAD_REF: ${{ github.event.pull_request.head.ref }} + run: | + set -euo pipefail + ref=dev + if [ -n "${HEAD_REF}" ] && git ls-remote --exit-code --heads "https://github.com/${SIBLING}.git" "refs/heads/${HEAD_REF}" >/dev/null 2>&1; then + ref="${HEAD_REF}" + fi + echo "ref=${ref}" >> "$GITHUB_OUTPUT" + echo "Comparing this pull request's corpus against ${SIBLING}@${ref}." + + - name: Checkout this mirror + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + path: mirror + + - name: Checkout the sibling mirror + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + repository: ${{ steps.mirror.outputs.sibling }} + ref: ${{ steps.sibling.outputs.ref }} + path: sibling + + # `README.md` is per-repo prose and the `*.fixture.ts` twins are generated by `mthds-js` + # alone; everything else in the tree is the shared artifact. Keep these exclusions in step + # with EXCLUDED_NAMES / EXCLUDED_SUFFIXES in conformance's parity script. + - name: Compare the two copies of the protocol fixture corpus + env: + SIBLING: ${{ steps.mirror.outputs.sibling }} + SIBLING_REF: ${{ steps.sibling.outputs.ref }} + run: | + set -euo pipefail + if diff --recursive --brief -x README.md -x '*.fixture.ts' mirror/tests/fixtures/protocol sibling/tests/fixtures/protocol; then + echo "OK: the protocol fixture corpus is byte-identical to ${SIBLING}@${SIBLING_REF}." + else + echo "::error::The protocol fixture corpus differs from ${SIBLING}@${SIBLING_REF}. A capture lands in both mirrors or in neither: regenerate it with \`pipelex-dev generate-projection-corpus\` in the pipelex checkout, copy the tree into both repos (see tests/fixtures/protocol/README.md), and give the paired pull request the same branch name as this one so this job compares against it instead of against dev." + exit 1 + fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 0631237..1172560 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## [Unreleased] +### Added + +- **A CI gate on the shared protocol fixture corpus.** `tests/fixtures/protocol/` is committed byte-identically here and in `mthds-js`, and until now nothing in either repo checked that: the only comparison lived in `conformance`, where it ran on a `conformance` pull request rather than on the one that moved the corpus. A recapture landed here without its `mthds-js` twin was therefore caught weeks later by an unrelated change, and the pairing held only because prose in two pull request bodies asked for it. `.github/workflows/protocol-corpus-parity.yml` now checks `mthds-js` out on every pull request and compares the two trees, so a one-sided recapture fails the pull request that causes it. + + It needs no secret — both mirrors are public, so the sibling checkout is unauthenticated and the job runs on a pull request from a fork exactly as it does on one from a branch. It compares against the sibling's branch of the same name when one exists and against `dev` otherwise, because a recapture lands as a paired pull request in both mirrors at once and comparing each against the other's `dev` would leave both red until the first merged. The file is a **vendored copy** of `conformance/tests/mthds/fixtures/protocol_corpus_parity_workflow.yml` and is byte-identical in both mirrors, which is why it derives which repo it is running in rather than naming one; `make check-fixture-drift` in `conformance` fails when a copy drifts or goes missing, so edit the canonical copy and re-vendor rather than editing this one in place. + ### Changed - **The projection fixture corpus covers the natives whose payload shape the descriptor states openly.** A new capture from `pipelex` `dev` adds `input_semantics_scaffold.scaffold_open_natives`, a pipe putting `native.JSON`, `native.Dynamic` and `native.Composite` each at a slot position — the one place the corpus never reached. `Dynamic` and `Composite` are structureless by design, so their node is `unknown` and both renderers agree on the empty mapping; `JSON` is not, because its pinned blueprint carries a required `json_obj`, which puts an unknown node at a **required object-field** position for the first time in the corpus — the required unknown nodes it already held sit at list-item positions, where no divergence was declared. That position is what the capture declares as a new divergence class, `unknown-empty-object`: the projection renders the empty object, because the descriptor withholds the payload shape there and inventing one would stop it projecting the descriptor, while the reference engine reflects the runtime content class and fills the dict with a sample key/value pair whoever fills the template in has to delete. The empty object round-trips through the input shaper cleanly, so the class records a difference of vantage rather than a defect and carries no ledger item. `input_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with it, and `mthds-js` commits the identical bytes. From 4ae3f1d718f30507324a3868972c858e4312391f Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Wed, 2 Sep 2026 16:53:48 +0200 Subject: [PATCH 04/11] changelog: re-file the output-form entries under Unreleased, not the published v0.12.0 (#61) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ab27956` added its three entries under `## [v0.12.0] - 2026-09-01`, a version published the day before it landed. The code they describe is not in that release: $ git cat-file -e v0.12.0:mthds/protocol/output_form.py -> absent $ git cat-file -e main:mthds/protocol/output_form.py -> absent So the released notes claimed a module and a contract change nobody who installed 0.12.0 has, while `[Unreleased]` — the section the next release draws from — carried neither. The breaking one is what makes this worth a commit of its own: `PipeOutputContract.json_schema` is a required member on a closed shape, so a payload from an earlier engine fails to parse, and cutting a release from `[Unreleased]` as it stood would have shipped that break with no note under its own heading. This is a pure move; no wording changes. The new module and its suite go to `### Added`, and the required member on an existing contract to `### Changed`, which is what each of them is. Two entries deliberately stay under v0.12.0. The inputs-template projection, the TOML emitter and the corpus were all in that release — `tests/fixtures/protocol/inputs_template/manifest.json` is present at the tag — so their attribution is correct. The longer restatement of the corpus entry that `ab27956` also added duplicates the shorter one above it, but it describes v0.12.0 content and is not misattributed, so it is left alone. Claude-Session: https://claude.ai/code/session_012E1hEFhVP2Ze394Aic1v58 Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1172560..a12a913 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,24 @@ ### Added +- **`mthds.protocol.output_form` — the output-form descriptor, mirroring the standard's new page.** `PipeOutputFormDescriptor` carries one `field`, not a `fields` list: a pipe has exactly one output where it may have many inputs, which is the only shape difference from its input twin and follows from the language. `OutputForm` is the `pipe_ref` → descriptor map, over the same key set as `pipe_io_contracts` and `input_form`. + + The module declares no node type. `field` is `InputFormField`, reused verbatim — an output is a concept reference exactly like an input is, so its kinds, its nesting and its constraints are the same questions with the same answers, and a second node union would be a second place for them to drift. What it adds is the enforcement of what an output does NOT carry: `presence` and `gating` are facts of an input slot, both already optional on the node so that a slotless node could exist, and a producer that fills them in with something plausible now fails the parse. The wire-null rule (`title: None` is absent, never `null`) is stated on the container for the same reason its input twin states it there — only at that level can a raw wire mapping be told from the keyword arguments of a programmatic construction. + +- **`tests/unit/test_protocol_output_form.py`** — parity against the new `output_form.json` capture, and the rules a type cannot state: the shared key set, the absence of both slot facts, and the **plural wrap** (a plural output described as a `list`, a single one not), which is the one place producing this artifact is real work rather than reuse and the one that fails silently. It also asserts what the corpus must reach — both plural arms — since a capture with no plural output could not have caught the wrap being skipped at all. + - **A CI gate on the shared protocol fixture corpus.** `tests/fixtures/protocol/` is committed byte-identically here and in `mthds-js`, and until now nothing in either repo checked that: the only comparison lived in `conformance`, where it ran on a `conformance` pull request rather than on the one that moved the corpus. A recapture landed here without its `mthds-js` twin was therefore caught weeks later by an unrelated change, and the pairing held only because prose in two pull request bodies asked for it. `.github/workflows/protocol-corpus-parity.yml` now checks `mthds-js` out on every pull request and compares the two trees, so a one-sided recapture fails the pull request that causes it. It needs no secret — both mirrors are public, so the sibling checkout is unauthenticated and the job runs on a pull request from a fork exactly as it does on one from a branch. It compares against the sibling's branch of the same name when one exists and against `dev` otherwise, because a recapture lands as a paired pull request in both mirrors at once and comparing each against the other's `dev` would leave both red until the first merged. The file is a **vendored copy** of `conformance/tests/mthds/fixtures/protocol_corpus_parity_workflow.yml` and is byte-identical in both mirrors, which is why it derives which repo it is running in rather than naming one; `make check-fixture-drift` in `conformance` fails when a copy drifts or goes missing, so edit the canonical copy and re-vendor rather than editing this one in place. ### Changed +- **`PipeOutputContract.json_schema` — the payload's schema (breaking).** The output contract was a closed shape with four members and no schema, on the reasoning that a run's payload is the run's own result. That answers *what did this run produce?*, correctly not a contract's business, rather than the question a consumer asks: *what shape will it be?* — which is declared in the source and knowable before any run. + + **It is the PAYLOAD's schema, and that is where its rule departs from the input twin's.** An input's describes what a caller *sends*, so a plural slot's is a bare array. An output's describes what *comes back*, which is the concept's content model — so a `native.Text` output's is the object declaring `text`, and a `Concept[]` output's is the list content model rather than a bare array. On the fixed arm the element array carries `minItems`/`maxItems` equal to `item_count`; since `optional` already states the output may be absent, the schema describes the shape when present and carries no null arm. + + Breaking because the shape is closed and the member is required: a payload from an earlier engine fails the parse, which is the intended behaviour for a minor version of the artifact. + - **The projection fixture corpus covers the natives whose payload shape the descriptor states openly.** A new capture from `pipelex` `dev` adds `input_semantics_scaffold.scaffold_open_natives`, a pipe putting `native.JSON`, `native.Dynamic` and `native.Composite` each at a slot position — the one place the corpus never reached. `Dynamic` and `Composite` are structureless by design, so their node is `unknown` and both renderers agree on the empty mapping; `JSON` is not, because its pinned blueprint carries a required `json_obj`, which puts an unknown node at a **required object-field** position for the first time in the corpus — the required unknown nodes it already held sit at list-item positions, where no divergence was declared. That position is what the capture declares as a new divergence class, `unknown-empty-object`: the projection renders the empty object, because the descriptor withholds the payload shape there and inventing one would stop it projecting the descriptor, while the reference engine reflects the runtime content class and fills the dict with a sample key/value pair whoever fills the template in has to delete. The empty object round-trips through the input shaper cleanly, so the class records a difference of vantage rather than a defect and carries no ledger item. `input_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with it, and `mthds-js` commits the identical bytes. - **The corpus suite notices a capture that arrives half-formed.** Two properties the record always had were never asserted, so either could break with the suite still green. The README beside the corpus explains each divergence class in prose, and that was the half nothing enforced: the manifest is checked hard, so a class that stops occurring has to be retired deliberately, but a retired class could leave its bullet behind describing a difference that no longer exists — and the suite, parametrised over the manifest's own list, would simply shrink rather than fail. `test_inputs_template_projection.py` now holds the README's bullet list to the manifest's declarations in both directions, the twin of the check `mthds-js` carries. It also asserts that `input_form.json` and `pipe_io_contracts.json` name the same pipes: they are one capture taken in one command, but nothing outside the contracts file's own parse test read it, so one half could lose a pipe the other still held. @@ -26,18 +38,6 @@ - Updated `README.md` and `docs/runners.md` to reference the new client-side inputs-template projection. -- **`mthds.protocol.output_form` — the output-form descriptor, mirroring the standard's new page.** `PipeOutputFormDescriptor` carries one `field`, not a `fields` list: a pipe has exactly one output where it may have many inputs, which is the only shape difference from its input twin and follows from the language. `OutputForm` is the `pipe_ref` → descriptor map, over the same key set as `pipe_io_contracts` and `input_form`. - - The module declares no node type. `field` is `InputFormField`, reused verbatim — an output is a concept reference exactly like an input is, so its kinds, its nesting and its constraints are the same questions with the same answers, and a second node union would be a second place for them to drift. What it adds is the enforcement of what an output does NOT carry: `presence` and `gating` are facts of an input slot, both already optional on the node so that a slotless node could exist, and a producer that fills them in with something plausible now fails the parse. The wire-null rule (`title: None` is absent, never `null`) is stated on the container for the same reason its input twin states it there — only at that level can a raw wire mapping be told from the keyword arguments of a programmatic construction. - -- **`PipeOutputContract.json_schema` — the payload's schema (breaking).** The output contract was a closed shape with four members and no schema, on the reasoning that a run's payload is the run's own result. That answers *what did this run produce?*, correctly not a contract's business, rather than the question a consumer asks: *what shape will it be?* — which is declared in the source and knowable before any run. - - **It is the PAYLOAD's schema, and that is where its rule departs from the input twin's.** An input's describes what a caller *sends*, so a plural slot's is a bare array. An output's describes what *comes back*, which is the concept's content model — so a `native.Text` output's is the object declaring `text`, and a `Concept[]` output's is the list content model rather than a bare array. On the fixed arm the element array carries `minItems`/`maxItems` equal to `item_count`; since `optional` already states the output may be absent, the schema describes the shape when present and carries no null arm. - - Breaking because the shape is closed and the member is required: a payload from an earlier engine fails the parse, which is the intended behaviour for a minor version of the artifact. - -- **`tests/unit/test_protocol_output_form.py`** — parity against the new `output_form.json` capture, and the rules a type cannot state: the shared key set, the absence of both slot facts, and the **plural wrap** (a plural output described as a `list`, a single one not), which is the one place producing this artifact is real work rather than reuse and the one that fails silently. It also asserts what the corpus must reach — both plural arms — since a capture with no plural output could not have caught the wrap being skipped at all. - - **The shared projection fixture corpus.** `tests/fixtures/protocol/inputs_template/` pairs each captured pipe's input-form descriptor with the fill-in inputs template a client-side projection must produce from it — the compact and explicit shapes, as JSON and as TOML — and `tests/unit/test_inputs_template_projection.py` holds the corpus to five properties: the whole closed `FieldKind` vocabulary is exercised, every pipe is present in both shapes and both formats, each deliberate departure from the reference engine's own renderer is still visible in the committed bytes, the manifest's `unshapeable` record stays keyed to this corpus, and (skip-gated until `mthds.protocol.inputs_template` lands) the projection reproduces every file byte for byte. That `unshapeable` array is the capture's own statement about which of its pinned templates the runtime's input shaper refuses today: the generator round-trips every projected template through `InputShaper.shape` and records each refusal with the error's class name and the ledger item whose fix retires it, so a template that cannot be filled in and handed back is declared rather than discovered. The corpus is committed identically in `mthds-js`, which runs the twin of that suite, and `conformance` compares the two mirrors file by file — that byte identity is the point, since the two projections are written in different languages and must agree exactly, TOML comment lines included. The descriptor and contract captures are regenerated from the same command and gain a third bundle covering a text field merely named `url` beside a real file position, optional file fields nested inside a structure, and both a fixed `[N]` and a variable `[]` slot over a structured concept, plus a fourth covering OUTPUTS — flat and nested structures, both plural arms, an optional output, a wrapping native scalar and a file-bearing one. That fourth bundle exists because the three before it all resolved to `Text`: a corpus whose every pipe outputs one native scalar can only capture one output descriptor and one output schema, and the two facts most likely to be got wrong would never appear in it. ## [v0.11.1] - 2026-08-28 From 58ba645de584728a2329bdd6cd119e1b3f4e0fde Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 6 Sep 2026 14:09:47 +0200 Subject: [PATCH 05/11] chore: re-capture the projection fixture corpus from pipelex dev (#63) Two pipes join the corpus from scaffold_bundle.mthds: scaffold_anything_slot, which places native.Anything at an input slot the descriptor states as an unknown node, and scaffold_dynamic_prompt, which pairs native.Dynamic with a structured concept whose text fields are named url and homepage_url. Both add worked sites to the existing text-named-url and optional-field-included divergence classes rather than declaring a new one, so the projection needed no change. The unshapeable record gains the scaffold_anything_slot pair in both shapes, refused with a StuffFactoryError and tracked by L-260902-10eb56. The fixtures README's provenance is corrected alongside: it listed three bundles where the capture has been taken from four since output_bundle.mthds joined, and omitted output_form.json from the files to copy across. Captured from pipelex dev at 551fe3463. mthds-js commits the identical bytes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AUyEZQcLJ4Qhb9vKoSw6vD --- CHANGELOG.md | 11 ++ tests/fixtures/protocol/README.md | 7 +- tests/fixtures/protocol/input_form.json | 61 +++++++++ ...affold.scaffold_anything_slot.compact.json | 6 + ...affold.scaffold_anything_slot.compact.toml | 2 + ...ffold.scaffold_anything_slot.explicit.json | 6 + ...ffold.scaffold_anything_slot.explicit.toml | 4 + ...ffold.scaffold_dynamic_prompt.compact.json | 12 ++ ...ffold.scaffold_dynamic_prompt.compact.toml | 4 + ...fold.scaffold_dynamic_prompt.explicit.json | 15 +++ ...fold.scaffold_dynamic_prompt.explicit.toml | 13 ++ .../protocol/inputs_template/manifest.json | 18 ++- tests/fixtures/protocol/output_form.json | 18 +++ .../fixtures/protocol/pipe_io_contracts.json | 125 ++++++++++++++++++ 14 files changed, 297 insertions(+), 5 deletions(-) create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.toml create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.json create mode 100644 tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.toml diff --git a/CHANGELOG.md b/CHANGELOG.md index 3633916..4ca3e72 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## [Unreleased] + +### Changed + +- **The projection fixture corpus covers `native.Anything` at a slot and a text field merely named like a URL.** A recapture from `pipelex` `dev` adds two pipes from `scaffold_bundle.mthds`: one placing `native.Anything` at an input slot, which the descriptor states as an `unknown` node, and one pairing `native.Dynamic` with a structured concept whose text fields are named `url` and `homepage_url`, which grows the worked sites of the existing `text-named-url` and `optional-field-included` classes without declaring a new one. `input_form.json`, `output_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with it, and `mthds-js` commits the identical bytes. +- **The unshapeable record names a second open engine bug.** Both shapes of the new `scaffold_anything_slot` template are refused by the runtime's input shaper with a `StuffFactoryError`, tracked by `L-260902-10eb56`: at a `native.Anything` slot the shaper accepts a bare string and nothing else, refusing the empty object that slot's own published contract gives as its template, and shapes even that string into a `native.Text` stuff rather than an `Anything` one. Until then the record's every entry belonged to the nested-list slot under `L-260830-191719`. + +### Fixed + +- **The fixture corpus README's provenance named the wrong capture.** It listed three bundles where the committed capture has been taken from four since `output_bundle.mthds` joined, and omitted `output_form.json` from the files to copy across, so following it reproduced neither the committed bytes nor a complete capture. + ## [v0.13.0] - 2026-09-02 ### Added diff --git a/tests/fixtures/protocol/README.md b/tests/fixtures/protocol/README.md index ca984a8..9ac7e2d 100644 --- a/tests/fixtures/protocol/README.md +++ b/tests/fixtures/protocol/README.md @@ -4,7 +4,7 @@ ## Provenance -- Bundles, in this order: `pipelex/tests/data/input_semantics/hinted_bundle.mthds`, `probe_bundle.mthds`, `scaffold_bundle.mthds`. The argument order is part of the capture: it decides the key order of the emitted maps, so a swapped order produces the same content with different bytes and breaks the byte parity with `mthds-js`. A bundle added at the end keeps the existing bytes stable. +- Bundles, in this order: `pipelex/tests/data/input_semantics/hinted_bundle.mthds`, `probe_bundle.mthds`, `scaffold_bundle.mthds`, `output_bundle.mthds`. The argument order is part of the capture: it decides the key order of the emitted maps, so a swapped order produces the same content with different bytes and breaks the byte parity with `mthds-js`. A bundle added at the end keeps the existing bytes stable. - Command, run at the `pipelex` checkout root: ```bash @@ -12,10 +12,11 @@ tests/data/input_semantics/hinted_bundle.mthds \ tests/data/input_semantics/probe_bundle.mthds \ tests/data/input_semantics/scaffold_bundle.mthds \ + tests/data/input_semantics/output_bundle.mthds \ -o ``` - Copy `input_form.json`, `pipe_io_contracts.json` and the whole `inputs_template/` tree from `` into this directory and into `mthds-js`'s twin of it, then run `npm run fixtures:protocol` there to regenerate its TypeScript twins. The `engine/` directory the command also writes is **not** committed: it holds the reference engine's own renderings, which is what the divergence record below is measured against. + Copy `input_form.json`, `output_form.json`, `pipe_io_contracts.json` and the whole `inputs_template/` tree from `` into this directory and into `mthds-js`'s twin of it, then run `npm run fixtures:protocol` there to regenerate its TypeScript twins. The `engine/` directory the command also writes is **not** committed: it holds the reference engine's own renderings, which is what the divergence record below is measured against. - Engine: `pipelex`, at the change that introduced `pipelex-dev generate-projection-corpus` (its own page is `pipelex/docs/contribute/generate-projection-corpus.md`). That command replaced `trace-input-semantics` as the producer of these files; generating from the two original bundles alone reproduces the previous capture byte for byte, so the move was a no-op diff. - Pages the models mirror: `mthds` v0.9.0, `docs/spec/pipe-io-contracts.md` and `docs/spec/input-form-descriptor.md` (the `hints` slot's shape from `docs/spec/intent-hints.md`), with the pinned native definitions from `docs/spec/native-concepts.md`. @@ -42,7 +43,7 @@ Each entry carries the workspace-ledger item tracking the engine fix, or `null` The templates are held to a second bar that is easy to lose sight of behind the byte parity: they must still **run**. A template is what someone fills in and hands back, so every slot of it has to survive the runtime's own input shaper. That is what separates a deliberate divergence from a projection bug — the file-leaf entry pins `{"url": ...}` because the shaper accepts exactly that wrapper, and the object-native entry keeps the envelope for the same reason. Where a class is retired, the corpus is regenerated and its entry disappears from the manifest on its own. -That bar is now measured rather than argued. The generator hands every projected template, in both shapes, to `InputShaper.shape` at capture time and writes the verdict into the manifest's `unshapeable` array — one entry per refused `(pipe_ref, shape)`, carrying the error's class name and the ledger item whose fix retires it — refusing to write a capture that holds a refusal nobody declared, or that declares one which has started shaping. Every entry this capture carries is blocked on `L-260830-191719`, the nested-list slot the shaper cannot take back. +That bar is now measured rather than argued. The generator hands every projected template, in both shapes, to `InputShaper.shape` at capture time and writes the verdict into the manifest's `unshapeable` array — one entry per refused `(pipe_ref, shape)`, carrying the error's class name and the ledger item whose fix retires it — refusing to write a capture that holds a refusal nobody declared, or that declares one which has started shaping. The entries this capture carries fall under two open engine bugs. The `probe_markers` and `probe_single` templates are blocked on `L-260830-191719`, the nested-list slot the shaper cannot take back. The `scaffold_anything_slot` pair is blocked on `L-260902-10eb56`: at a `native.Anything` input slot the shaper accepts a bare string and nothing else — refusing the empty object that slot's own published contract gives as its template — and even that string it shapes into a `native.Text` stuff rather than an `Anything` one. The array is a statement the corpus makes about itself, and this repo cannot re-derive it: there is no input shaper on this side of the mirror, so the verdict is taken on the generator's authority. What `TestTheUnshapeableRecord` checks is that the record stays about *this* corpus — every entry keyed to a pipe and shape the manifest holds, one entry per key, each naming a real error type and a well-formed ledger id, and the array an exception list rather than the whole corpus. A consumer harness may read the entries to know which pinned templates it must not expect to run; nothing requires it to. diff --git a/tests/fixtures/protocol/input_form.json b/tests/fixtures/protocol/input_form.json index 5919277..e6f752d 100644 --- a/tests/fixtures/protocol/input_form.json +++ b/tests/fixtures/protocol/input_form.json @@ -1484,6 +1484,67 @@ } ] }, + "input_semantics_scaffold.scaffold_anything_slot": { + "fields": [ + { + "kind": "unknown", + "name": "anything_in", + "concept_ref": "native.Anything", + "description": "Anything", + "required": true, + "presence": "plain", + "gating": true + } + ] + }, + "input_semantics_scaffold.scaffold_dynamic_prompt": { + "fields": [ + { + "kind": "unknown", + "name": "dynamic_prompted", + "concept_ref": "native.Dynamic", + "description": "A dynamic concept", + "required": true, + "presence": "plain", + "gating": true + }, + { + "kind": "object", + "name": "bookmark", + "concept_ref": "input_semantics_scaffold.Bookmark", + "description": "SCAFFOLD_desc_concept_Bookmark: a shelf reference whose text fields are named like links", + "required": true, + "presence": "plain", + "gating": true, + "fields": [ + { + "kind": "text", + "name": "url", + "description": "SCAFFOLD_desc_field_url: a required text field merely NAMED url", + "required": true + }, + { + "kind": "text", + "name": "homepage_url", + "description": "SCAFFOLD_desc_field_homepage_url: an optional text field whose name ENDS in _url", + "required": false + }, + { + "kind": "text", + "name": "label", + "description": "SCAFFOLD_desc_field_label: ordinary required text beside them", + "required": true + }, + { + "kind": "text", + "name": "shelf_note", + "description": "SCAFFOLD_desc_field_shelf_note: ordinary optional text beside them", + "required": false + } + ] + } + ] + }, "input_semantics_output.output_structured": { "fields": [ { diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.json new file mode 100644 index 0000000..bde10ec --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.json @@ -0,0 +1,6 @@ +{ + "anything_in": { + "concept": "native.Anything", + "content": {} + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.toml new file mode 100644 index 0000000..00e03a9 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.compact.toml @@ -0,0 +1,2 @@ +# concept: native.Anything +anything_in = {concept = "native.Anything", content = {}} diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.json new file mode 100644 index 0000000..bde10ec --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.json @@ -0,0 +1,6 @@ +{ + "anything_in": { + "concept": "native.Anything", + "content": {} + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.toml new file mode 100644 index 0000000..3d09248 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_anything_slot.explicit.toml @@ -0,0 +1,4 @@ +[anything_in] +concept = "native.Anything" + +[anything_in.content] diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.json new file mode 100644 index 0000000..620378c --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.json @@ -0,0 +1,12 @@ +{ + "dynamic_prompted": { + "concept": "native.Dynamic", + "content": {} + }, + "bookmark": { + "url": "url_value", + "homepage_url": "homepage_url_value", + "label": "label_value", + "shelf_note": "shelf_note_value" + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.toml new file mode 100644 index 0000000..714ce5e --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.compact.toml @@ -0,0 +1,4 @@ +# concept: native.Dynamic +dynamic_prompted = {concept = "native.Dynamic", content = {}} +# concept: input_semantics_scaffold.Bookmark +bookmark = {url = "url_value", homepage_url = "homepage_url_value", label = "label_value", shelf_note = "shelf_note_value"} diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.json b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.json new file mode 100644 index 0000000..1a2ca47 --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.json @@ -0,0 +1,15 @@ +{ + "dynamic_prompted": { + "concept": "native.Dynamic", + "content": {} + }, + "bookmark": { + "concept": "input_semantics_scaffold.Bookmark", + "content": { + "url": "url_value", + "homepage_url": "homepage_url_value", + "label": "label_value", + "shelf_note": "shelf_note_value" + } + } +} \ No newline at end of file diff --git a/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.toml b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.toml new file mode 100644 index 0000000..a25893e --- /dev/null +++ b/tests/fixtures/protocol/inputs_template/input_semantics_scaffold.scaffold_dynamic_prompt.explicit.toml @@ -0,0 +1,13 @@ +[dynamic_prompted] +concept = "native.Dynamic" + +[dynamic_prompted.content] + +[bookmark] +concept = "input_semantics_scaffold.Bookmark" + +[bookmark.content] +url = "url_value" +homepage_url = "homepage_url_value" +label = "label_value" +shelf_note = "shelf_note_value" diff --git a/tests/fixtures/protocol/inputs_template/manifest.json b/tests/fixtures/protocol/inputs_template/manifest.json index c8ee5d6..c17768e 100644 --- a/tests/fixtures/protocol/inputs_template/manifest.json +++ b/tests/fixtures/protocol/inputs_template/manifest.json @@ -18,6 +18,8 @@ "input_semantics_probe.probe_native_inputs", "input_semantics_probe.probe_refined", "input_semantics_probe.probe_single", + "input_semantics_scaffold.scaffold_anything_slot", + "input_semantics_scaffold.scaffold_dynamic_prompt", "input_semantics_scaffold.scaffold_fixed_batch", "input_semantics_scaffold.scaffold_nesting", "input_semantics_scaffold.scaffold_open_natives" @@ -236,7 +238,7 @@ "divergence_id": "optional-field-included", "reason": "The engine passes include_optional=False at the top of an input's own structure class and not through the recursion, so it hides an optional field at depth one and shows one nested deeper. The projection renders every field the descriptor states, at every depth.", "ledger_item": "L-260830-d51440", - "occurrences": 163, + "occurrences": 167, "examples": [ { "pipe_ref": "input_semantics_hinted.hinted_slots", @@ -265,7 +267,7 @@ "divergence_id": "text-named-url", "reason": "The engine picks a placeholder by field name, so a text field merely named url or ending in _url renders as a URL. The projection reads the descriptor's kind instead.", "ledger_item": "L-260830-dc48bf", - "occurrences": 4, + "occurrences": 6, "examples": [ { "pipe_ref": "input_semantics_scaffold.scaffold_nesting", @@ -341,6 +343,18 @@ "shape": "explicit", "error_type": "ValidationError", "ledger_item": "L-260830-191719" + }, + { + "pipe_ref": "input_semantics_scaffold.scaffold_anything_slot", + "shape": "compact", + "error_type": "StuffFactoryError", + "ledger_item": "L-260902-10eb56" + }, + { + "pipe_ref": "input_semantics_scaffold.scaffold_anything_slot", + "shape": "explicit", + "error_type": "StuffFactoryError", + "ledger_item": "L-260902-10eb56" } ] } diff --git a/tests/fixtures/protocol/output_form.json b/tests/fixtures/protocol/output_form.json index a460752..6dcad97 100644 --- a/tests/fixtures/protocol/output_form.json +++ b/tests/fixtures/protocol/output_form.json @@ -71,6 +71,24 @@ "required": true } }, + "input_semantics_scaffold.scaffold_anything_slot": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, + "input_semantics_scaffold.scaffold_dynamic_prompt": { + "field": { + "kind": "prose", + "name": "output", + "concept_ref": "native.Text", + "description": "A text", + "required": true + } + }, "input_semantics_output.output_structured": { "field": { "kind": "object", diff --git a/tests/fixtures/protocol/pipe_io_contracts.json b/tests/fixtures/protocol/pipe_io_contracts.json index 2ad66d0..db0fd8a 100644 --- a/tests/fixtures/protocol/pipe_io_contracts.json +++ b/tests/fixtures/protocol/pipe_io_contracts.json @@ -3565,6 +3565,131 @@ } } }, + "input_semantics_scaffold.scaffold_anything_slot": { + "inputs": { + "anything_in": { + "concept_ref": "native.Anything", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "title": "native.Anything", + "description": "Anything" + } + } + }, + "output": { + "concept_ref": "native.Text", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, + "input_semantics_scaffold.scaffold_dynamic_prompt": { + "inputs": { + "dynamic_prompted": { + "concept_ref": "native.Dynamic", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "properties": {}, + "title": "native.Dynamic", + "type": "object", + "description": "A dynamic concept" + } + }, + "bookmark": { + "concept_ref": "input_semantics_scaffold.Bookmark", + "presence": "plain", + "multiplicity": "single", + "item_count": null, + "json_schema": { + "description": "SCAFFOLD_desc_concept_Bookmark: a shelf reference whose text fields are named like links", + "properties": { + "url": { + "description": "SCAFFOLD_desc_field_url: a required text field merely NAMED url", + "title": "Url", + "type": "string" + }, + "homepage_url": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "SCAFFOLD_desc_field_homepage_url: an optional text field whose name ENDS in _url", + "title": "Homepage Url" + }, + "label": { + "description": "SCAFFOLD_desc_field_label: ordinary required text beside them", + "title": "Label", + "type": "string" + }, + "shelf_note": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "SCAFFOLD_desc_field_shelf_note: ordinary optional text beside them", + "title": "Shelf Note" + } + }, + "required": [ + "url", + "label" + ], + "title": "input_semantics_scaffold.Bookmark", + "type": "object" + } + } + }, + "output": { + "concept_ref": "native.Text", + "multiplicity": "single", + "item_count": null, + "optional": false, + "json_schema": { + "description": "A text", + "properties": { + "text": { + "description": "The text", + "title": "Text", + "type": "string" + } + }, + "required": [ + "text" + ], + "title": "native.Text", + "type": "object" + } + } + }, "input_semantics_output.output_structured": { "inputs": { "note": { From 822baad7108bca8c003fac4bbc25a0377bf87250 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sun, 6 Sep 2026 23:03:10 +0200 Subject: [PATCH 06/11] =?UTF-8?q?feature:=20follow=20the=20standard's=20ve?= =?UTF-8?q?rsion=20cut=20=E2=80=94=20MTHDS=5FSTANDARD=5FVERSION=202.0.0,?= =?UTF-8?q?=20each=20constant=20declared=20once=20(#64)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feature: follow the standard's version cut — MTHDS_STANDARD_VERSION 2.0.0, each constant declared once The MTHDS specification unified its release number with its standard version and cut both at 2.0.0 (mthds#63), a major accounting for the breaking native, manifest, lock and resolution changes it had shipped while MTHDS_STANDARD_VERSION stood still at 1.0.0 since February. This package copies that cut: the standard version becomes 2.0.0, the protocol version stays 0.6.0 — the two numbers are independent and the routes did not change — and each constant now carries a docstring naming https://mthds.ai/spec/versioning/ as the rule that governs it, so following the next cut is one edit per number rather than a search. Adds `is_mthds_version_satisfied`, which evaluates a manifest's `mthds_version` against MTHDS_STANDARD_VERSION and against nothing else. The constraint was validated for spelling here and then evaluated nowhere, so every runtime open-coded the parse-and-match itself. Parsing a manifest still does not evaluate it: a package asking for a standard version this runtime does not implement is a well-formed manifest, and whether that warns, refuses the load or fails validation is the runtime's call. Fixes a pre-existing bug found at that seam: `parse_constraint` handed the string to SimpleSpec untouched, which rejects ">=1.0.0, <2.0.0" over the space alone — while the manifest format documents that exact spelling and `is_valid_version_constraint` accepts it. Any manifest or dependency written that way raised SemVerError from the resolver instead of resolving. The `PipelexRunner.version` TODO stays; it retires when the pipelex CLI exposes `protocol_version` (L-260829-43c082). Adds docs/versioning.md. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_017VEgU3Bn7kfRA4ddJZ48HH * fix: prefix every spec-site link with /latest/, and pin the constraint spellings mthds.ai publishes each release under its own prefix and redirects only the root to /latest/, so a bare https://mthds.ai/spec// is a 404 while https://mthds.ai/latest/spec// is the page. Verified live: the bare shape returns 404 for input-form-descriptor and pipe-io-contracts, the prefixed shape returns 200 for both. Every link into the site — the docstrings this branch adds, docs/versioning.md, and the ones in docs/inputs-template.md and docs/runners.md that predate it — now carries the prefix. The versioning page is the exception, and it stays a forward reference: it landed on the standard's dev branch in mthds-ai/mthds#63 and has never been published, so it is a 404 at both shapes. mthds.ai's `latest` still aliases 0.4.0, cut before the page existed. Publishing it is L-260906-50cf25. Also pins that is_valid_version_constraint and parse_constraint agree on every constraint spelling, in both directions. Stripping the blocks fixed the spaced compound constraint, and the risk it introduced is the opposite error: an empty block left by a doubled or trailing comma must stay a refusal rather than being silently dropped, which would widen the constraint the manifest asked for. Advances L-260829-5b9a42 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_019dKCMzkj6yhetfgNeA4cEm --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 8 ++++++ README.md | 1 + docs/inputs-template.md | 2 +- docs/runners.md | 2 +- docs/versioning.md | 38 +++++++++++++++++++++++++++ mthds/package/manifest/schema.py | 45 ++++++++++++++++++++++++++++++-- mthds/package/semver.py | 9 +++++-- mthds/protocol/protocol.py | 15 ++++++++--- mthds/runners/pipelex/runner.py | 7 ++--- tests/unit/test_semver.py | 32 ++++++++++++++++++++++- tests/unit/test_versions.py | 43 ++++++++++++++++++++++++++++++ 11 files changed, 188 insertions(+), 14 deletions(-) create mode 100644 docs/versioning.md create mode 100644 tests/unit/test_versions.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 4ca3e72..fc8a3e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,13 +2,21 @@ ## [Unreleased] +### Added + +- **`is_mthds_version_satisfied` — a manifest's `mthds_version` evaluated against the standard version, and against nothing else.** The constraint was validated for spelling and then never evaluated here, so every runtime open-coded the parse-and-match against `MTHDS_STANDARD_VERSION` itself. Parsing a manifest still does not evaluate it: a package asking for a standard version this runtime does not implement is a well-formed manifest, and whether that warns, refuses the load or fails validation is the runtime's call. +- **`docs/versioning.md`** — the four numbers in play when you use this package, which two are copies of the standard's cut, and why a manifest constraint and a crate stamp mean different things while sharing one number. + ### Changed +- **`MTHDS_STANDARD_VERSION` follows the standard's cut: `1.0.0` → `2.0.0`.** The MTHDS specification unified its release number and its standard version and cut both at `2.0.0`, a major accounting for the breaking native, manifest, lock and resolution changes it had already shipped while the constant stood still ([the rule](https://mthds.ai/latest/spec/versioning/)). This package copies that cut. Consumers feel it in two places: a manifest constraint is now evaluated against `2.0.0` (a `>=1.0.0` stays satisfied, a `^1.0.0` no longer is), and `pipelex` stamps the crates it normalizes with the new version. `PROTOCOL_VERSION` stays `0.6.0` — the protocol is versioned separately and the routes did not change. Both constants now carry a docstring naming the spec page that governs them, since each is a copy of a cut rather than a decision this package makes. **(Breaking)** - **The projection fixture corpus covers `native.Anything` at a slot and a text field merely named like a URL.** A recapture from `pipelex` `dev` adds two pipes from `scaffold_bundle.mthds`: one placing `native.Anything` at an input slot, which the descriptor states as an `unknown` node, and one pairing `native.Dynamic` with a structured concept whose text fields are named `url` and `homepage_url`, which grows the worked sites of the existing `text-named-url` and `optional-field-included` classes without declaring a new one. `input_form.json`, `output_form.json`, `pipe_io_contracts.json` and `inputs_template/manifest.json` move with it, and `mthds-js` commits the identical bytes. - **The unshapeable record names a second open engine bug.** Both shapes of the new `scaffold_anything_slot` template are refused by the runtime's input shaper with a `StuffFactoryError`, tracked by `L-260902-10eb56`: at a `native.Anything` slot the shaper accepts a bare string and nothing else, refusing the empty object that slot's own published contract gives as its template, and shapes even that string into a `native.Text` stuff rather than an `Anything` one. Until then the record's every entry belonged to the nested-list slot under `L-260830-191719`. ### Fixed +- **A compound version constraint spelled with a space was unparsable.** `parse_constraint` handed the string to `SimpleSpec` untouched, which rejects `">=1.0.0, <2.0.0"` over the space alone — while the manifest format documents that exact spelling and `is_valid_version_constraint` accepts it. Any manifest or dependency written that way raised `SemVerError` from the resolver instead of resolving. The blocks are now stripped before parsing. +- **Every link into the specification site pointed at a URL shape it does not serve.** mthds.ai publishes each release under its own prefix and redirects the root to `/latest/`, but it rewrites nothing else, so a bare `https://mthds.ai/spec//` is a 404 while `https://mthds.ai/latest/spec//` is the page. Every such link — in the docstrings, in `docs/`, and the ones that predate this release — now carries the prefix. The versioning page itself stays a forward reference until the standard publishes a release past `0.4.0`, which `latest` still aliases, tracked by `L-260906-50cf25`. - **The fixture corpus README's provenance named the wrong capture.** It listed three bundles where the committed capture has been taken from four since `output_bundle.mthds` joined, and omitted `output_form.json` from the files to copy across, so following it reproduced neither the committed bytes nor a complete capture. ## [v0.13.0] - 2026-09-02 diff --git a/README.md b/README.md index 03ea846..aaaa95f 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ pip install mthds See [docs/runners.md](./docs/runners.md) for the protocol + runners reference. See [docs/inputs-template.md](./docs/inputs-template.md) for the inputs-template projection. +See [docs/versioning.md](./docs/versioning.md) for the MTHDS standard and protocol versions this package carries. ## Quick start diff --git a/docs/inputs-template.md b/docs/inputs-template.md index 7e9a688..9f82580 100644 --- a/docs/inputs-template.md +++ b/docs/inputs-template.md @@ -1,6 +1,6 @@ # The inputs-template projection -A pipe's **inputs template** is the fill-in document somebody hands back as that pipe's inputs: a person at a form, or an agent preparing a run. `mthds.protocol.inputs_template` projects one from the [input-form descriptor](https://mthds.ai/spec/input-form-descriptor/) the standard already defines, so a client holding the descriptor can offer a template for a method it does **not** have on disk — which is exactly what a `method_ref` address or a hosted `method_id` names. +A pipe's **inputs template** is the fill-in document somebody hands back as that pipe's inputs: a person at a form, or an agent preparing a run. `mthds.protocol.inputs_template` projects one from the [input-form descriptor](https://mthds.ai/latest/spec/input-form-descriptor/) the standard already defines, so a client holding the descriptor can offer a template for a method it does **not** have on disk — which is exactly what a `method_ref` address or a hosted `method_id` names. ```python from mthds.protocol.input_form import InputForm diff --git a/docs/runners.md b/docs/runners.md index 49f3e6d..6d3aca1 100644 --- a/docs/runners.md +++ b/docs/runners.md @@ -66,7 +66,7 @@ The Dict wire models are extension-open at every level, matching the protocol's ### The validate artifacts: `pipe_io_contracts` and `input_form` -Two artifacts of the valid `/validate` report are owned by the standard since `mthds` v0.9.0 — [Pipe I/O Contracts](https://mthds.ai/spec/pipe-io-contracts/) and the [Input-Form Descriptor](https://mthds.ai/spec/input-form-descriptor/) — and this package types them: `mthds/protocol/pipe_io_contracts.py` and `mthds/protocol/input_form.py` mirror those pages exactly, snake_case slot for snake_case slot. Both are **recommended extension fields** of the report, not base fields: the protocol's base did not change, and how a caller asks a runner for the descriptor is implementation-defined (the hosted Pipelex API gates it behind its `views` request extension). They therefore arrive beside the report's typed base fields, and you narrow them by declaring them as typed fields on a model that extends the report — pydantic parses the maps and the discriminated union from the plain annotations, with no adapter machinery (this is exactly how `pipelex-sdk`'s report narrowing consumes them): +Two artifacts of the valid `/validate` report are owned by the standard since `mthds` v0.9.0 — [Pipe I/O Contracts](https://mthds.ai/latest/spec/pipe-io-contracts/) and the [Input-Form Descriptor](https://mthds.ai/latest/spec/input-form-descriptor/) — and this package types them: `mthds/protocol/pipe_io_contracts.py` and `mthds/protocol/input_form.py` mirror those pages exactly, snake_case slot for snake_case slot. Both are **recommended extension fields** of the report, not base fields: the protocol's base did not change, and how a caller asks a runner for the descriptor is implementation-defined (the hosted Pipelex API gates it behind its `views` request extension). They therefore arrive beside the report's typed base fields, and you narrow them by declaring them as typed fields on a model that extends the report — pydantic parses the maps and the discriminated union from the plain annotations, with no adapter machinery (this is exactly how `pipelex-sdk`'s report narrowing consumes them): ```python from mthds.protocol.input_form import InputForm, ListField diff --git a/docs/versioning.md b/docs/versioning.md new file mode 100644 index 0000000..9521b74 --- /dev/null +++ b/docs/versioning.md @@ -0,0 +1,38 @@ +# Versions this package carries + +Four different numbers are in play when you use `mthds`, and only two of them are the standard's. Confusing them is what let the standard version sit still for six months while the language underneath it changed, so this page says what each one is and where it is written. + +| Number | Declared at | What it versions | +|---|---|---| +| **MTHDS standard version** | `MTHDS_STANDARD_VERSION`, `mthds/package/manifest/schema.py` | The language, the native concept set, the `METHODS.toml` and `methods.lock` formats, the library crate format, the namespace resolution rules | +| **MTHDS Protocol version** | `PROTOCOL_VERSION`, `mthds/protocol/protocol.py` | The HTTP runner contract — its routes and the shapes they exchange | +| This package's release | `pyproject.toml` | This library, and nothing else | +| A runner's version | `VersionInfo.runner_version`, reported by the runner | That runner's own build | + +The first two are **copies of a cut made by the standard**, not decisions this package makes. The rule that moves them — what counts as a major, a minor and a patch for each — is [`https://mthds.ai/latest/spec/versioning/`](https://mthds.ai/latest/spec/versioning/), and each constant is declared once, in one place, with that page named in its docstring. Following a cut is therefore a single edit per number. + +Two properties of the rule are worth knowing before you read a version out of this package: + +- **The two numbers are independent.** The protocol version never tracks the standard version. A release of the standard that leaves the routes alone leaves `PROTOCOL_VERSION` exactly where it is, which is why `0.6.0` and `2.0.0` are as far apart as they are. +- **The standard version moves on every release of the specification**, including one that changes nothing normative. So the number rising is not by itself evidence that anything you depend on changed; the standard's changelog is. The corollary is that the pinned native set is identified by the standard version in which it *last changed*, and an implementation of version `V` materializes the set pinned at the greatest version less than or equal to `V`. + +## `mthds_version` in a manifest is a constraint + +A package's `METHODS.toml` may declare a `mthds_version`. It is a constraint — the versions of the standard the package says it works with — and it is evaluated against `MTHDS_STANDARD_VERSION` and nothing else: never against this package's release, a runner's version, or the protocol version. + +```python +from mthds.package.manifest.parser import parse_methods_toml +from mthds.package.manifest.schema import is_mthds_version_satisfied + +manifest = parse_methods_toml(methods_toml_text) +if manifest.mthds_version is not None and not is_mthds_version_satisfied(manifest.mthds_version): + ... # this runtime implements a standard version the package does not accept +``` + +Parsing a manifest deliberately does **not** evaluate the constraint. A package asking for a standard version this runtime does not implement is a well-formed manifest, and what to do about it — warn, refuse to load it, fail validation — belongs to the runtime, which is why the evaluation is a helper you call rather than something `parse_methods_toml` decides for you. The parser checks only that the constraint is *spellable*: `is_valid_version_constraint` accepts the range syntax the manifest format documents (`2.0.0`, `^2.0.0`, `~2.0.0`, `>=2.0.0`, `>=2.0.0, <3.0.0`, `*`, `2.*`), and `is_mthds_version_satisfied` raises `SemVerError` on anything it cannot parse. + +The same number appears in a library crate as a **stamp** rather than a constraint: the exact standard version the crate was normalized against, recording which pinned native set produced its materialized natives. Crates are produced by the runtime (`pipelex`), which stamps them with this package's `MTHDS_STANDARD_VERSION`. + +## Reading a runner's versions + +`VersionInfo`, returned by a runner's `version()` route, carries `protocol_version` and an optional `runner_version`. The protocol version a runner reports is the one it implements — compare it against `PROTOCOL_VERSION` to know whether this client and that runner agree on the wire contract. `runner_version` is that server's own build number and means nothing outside it. diff --git a/mthds/package/manifest/schema.py b/mthds/package/manifest/schema.py index 047a15c..c0f7bd4 100644 --- a/mthds/package/manifest/schema.py +++ b/mthds/package/manifest/schema.py @@ -1,10 +1,11 @@ import re import unicodedata -from typing import Any, Self, cast +from typing import Any, Final, Self, cast from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator from mthds.package.manifest.validation import is_domain_code_valid, is_pipe_code_valid +from mthds.package.semver import parse_constraint, parse_version, version_satisfies # Semver regex: MAJOR.MINOR.PATCH with optional pre-release and build metadata SEMVER_PATTERN = re.compile( @@ -32,7 +33,21 @@ RESERVED_DOMAINS: frozenset[str] = frozenset({"native", "mthds", "pipelex"}) -MTHDS_STANDARD_VERSION: str = "1.0.0" +MTHDS_STANDARD_VERSION: Final[str] = "2.0.0" +"""The MTHDS standard version this library implements. + +This is a copy of a cut made by the standard, and the only place this library +states it. One number versions the language, the native concept set, the +`METHODS.toml` and `methods.lock` formats, the library crate format and the +namespace resolution rules — see https://mthds.ai/latest/spec/versioning/ ("The +Standard Version") for what bumps it, and for the rule that resolves the pinned +native set an implementation of this version materializes. + +Two artifacts carry it and mean different things: a manifest's `mthds_version` +is a *constraint*, evaluated against this value by +`is_mthds_version_satisfied`, and a library crate's `mthds_version` is a *stamp* +of the value the crate was normalized against. +""" # Method name: snake_case, 2-25 chars, must start with a letter METHOD_NAME_PATTERN = re.compile(r"^[a-z][a-z0-9_]{1,24}$") @@ -68,6 +83,32 @@ def is_valid_version_constraint(constraint: str) -> bool: return VERSION_CONSTRAINT_PATTERN.match(constraint.strip()) is not None +def is_mthds_version_satisfied(constraint: str) -> bool: + """Check whether the standard version this library implements satisfies a manifest constraint. + + A manifest's `mthds_version` is evaluated against `MTHDS_STANDARD_VERSION` + and against nothing else — never against the release version of this + package, of a runner, or of the protocol, which are separate numbers moving + on separate cadences (https://mthds.ai/latest/spec/versioning/). + + Evaluating the constraint is deliberately not part of parsing a manifest: a + package declaring a standard version this runtime does not implement is a + well-formed manifest, and what to do about it — warn, refuse to load, refuse + to validate — is the runtime's call. + + Args: + constraint: The `mthds_version` constraint from a manifest (e.g. ">=2.0.0"). + + Returns: + True if the implemented standard version satisfies the constraint. + + Raises: + SemVerError: If the constraint is not a parsable version constraint. + """ + parsed_constraint = parse_constraint(constraint) + return version_satisfies(parse_version(MTHDS_STANDARD_VERSION), constraint=parsed_constraint) + + def is_valid_address(address: str) -> bool: """Check if an address contains at least one dot before a slash (hostname pattern).""" return ADDRESS_PATTERN.match(address) is not None diff --git a/mthds/package/semver.py b/mthds/package/semver.py index fbda690..4c41cfd 100644 --- a/mthds/package/semver.py +++ b/mthds/package/semver.py @@ -40,8 +40,12 @@ def parse_version(version_str: str) -> Version: def parse_constraint(constraint_str: str) -> SimpleSpec: """Parse a constraint string into a semantic_version.SimpleSpec. + Whitespace around the blocks of a compound constraint is stripped first: + `SimpleSpec` rejects `">=1.0.0, <2.0.0"` over the space alone, while that is + the spelling the MTHDS manifest format documents and accepts. + Args: - constraint_str: The constraint string to parse (e.g. "^1.2.3", ">=1.0.0,<2.0.0"). + constraint_str: The constraint string to parse (e.g. "^1.2.3", ">=1.0.0, <2.0.0"). Returns: The parsed SimpleSpec object. @@ -49,8 +53,9 @@ def parse_constraint(constraint_str: str) -> SimpleSpec: Raises: SemVerError: If the constraint string is not valid. """ + normalized = ",".join(block.strip() for block in constraint_str.split(",")) try: - return SimpleSpec(constraint_str) + return SimpleSpec(normalized) except ValueError as exc: msg = f"Invalid semver constraint: {constraint_str!r}" raise SemVerError(msg) from exc diff --git a/mthds/protocol/protocol.py b/mthds/protocol/protocol.py index 5810c76..ca7afc4 100644 --- a/mthds/protocol/protocol.py +++ b/mthds/protocol/protocol.py @@ -18,10 +18,17 @@ PROTOCOL_VERSION: Final[str] = "0.6.0" """The MTHDS Protocol version this SDK implements. -Single source of truth. Every runner reports exactly this value as -`protocol_version` in its `version()` handshake — runners do not get to -override or interpret it. The protocol version is a property of the -standard, not of any individual runner. +This is a copy of a cut made by the standard, and the only place this library +states it. Every runner reports exactly this value as `protocol_version` in its +`version()` handshake — runners do not get to override or interpret it, because +the protocol version is a property of the standard rather than of any individual +runner. + +It versions the HTTP runner contract only — the routes and the shapes they +exchange — and it moves on its own cadence: a release of the standard that +leaves those alone leaves this number exactly where it is, so it never tracks +`MTHDS_STANDARD_VERSION`. See https://mthds.ai/latest/spec/versioning/ ("The Protocol +Version") for what bumps it. """ diff --git a/mthds/runners/pipelex/runner.py b/mthds/runners/pipelex/runner.py index 51c5466..0a4f0a4 100644 --- a/mthds/runners/pipelex/runner.py +++ b/mthds/runners/pipelex/runner.py @@ -379,9 +379,10 @@ async def version(self) -> VersionInfo: """ # TODO(pipelex): pipelex CLI must expose `protocol_version` for us to # report it here (e.g. extend `pipelex --version` to emit both the - # runner version and the MTHDS protocol version it implements). The - # value lives in pipelex as `MTHDS_PROTOCOL_VERSION` but is not - # surfaced through the CLI today. + # runner version and the MTHDS protocol version it implements). pipelex + # holds the value already — it imports `PROTOCOL_VERSION` from this + # package — but does not surface it through the CLI. Tracked as + # L-260829-43c082; this raise goes away when that lands. msg = ( "PipelexRunner cannot report `protocol_version`: the pipelex CLI does not yet expose it. " "Extend the pipelex CLI to emit the MTHDS protocol version it implements, then parse it here." diff --git a/tests/unit/test_semver.py b/tests/unit/test_semver.py index 7b63088..3cdca6a 100644 --- a/tests/unit/test_semver.py +++ b/tests/unit/test_semver.py @@ -1,6 +1,7 @@ import pytest from semantic_version import Version # type: ignore[import-untyped] +from mthds.package.manifest.schema import is_valid_version_constraint from mthds.package.semver import ( SemVerError, parse_constraint, @@ -37,16 +38,45 @@ def test_parse_version_invalid(self, invalid: str): # --- parse_constraint --- - @pytest.mark.parametrize("constraint_str", ["^1.0.0", ">=1.0.0,<2.0.0", "*"]) + @pytest.mark.parametrize("constraint_str", ["^1.0.0", ">=1.0.0,<2.0.0", ">=1.0.0, <2.0.0", " ^1.0.0 ", "*"]) def test_parse_constraint(self, constraint_str: str): result = parse_constraint(constraint_str) assert result is not None + def test_parse_constraint_ignores_whitespace_between_blocks(self): + # The manifest format documents the spaced spelling, and SimpleSpec rejects it over the space alone. + spaced = parse_constraint(">=1.0.0, <2.0.0") + assert version_satisfies(parse_version("1.5.0"), spaced) + assert not version_satisfies(parse_version("2.0.0"), spaced) + @pytest.mark.parametrize("invalid", [">>>1.0.0", "not_valid"]) def test_parse_constraint_invalid(self, invalid: str): with pytest.raises(SemVerError, match="Invalid semver constraint"): parse_constraint(invalid) + @pytest.mark.parametrize( + ("topic", "constraint_str", "is_parsable"), + [ + ("a tab between the blocks of a compound constraint", ">=1.0.0,\t<2.0.0", True), + ("a newline between them", ">=1.0.0,\n<2.0.0", True), + ("whitespace wrapping the whole constraint", "\n>=1.0.0\n", True), + ("an empty block left by a doubled comma", ">=1.0.0,,<2.0.0", False), + ("an empty block left by a trailing comma", ">=1.0.0, ", False), + ("whitespace inside a block, between the operator and the version", "> = 1.0.0", False), + ], + ) + def test_a_spelling_parses_exactly_when_the_manifest_format_accepts_it(self, topic: str, constraint_str: str, is_parsable: bool): + # The bug this guards: `is_valid_version_constraint` accepted a spelling that `parse_constraint` + # then refused, so a well-formed manifest raised SemVerError from the resolver instead of resolving. + # Stripping the blocks must never make the two disagree in either direction — in particular an empty + # block must stay a refusal rather than being silently dropped, which would widen the constraint. + assert is_valid_version_constraint(constraint_str) is is_parsable, topic + if is_parsable: + assert parse_constraint(constraint_str) is not None, topic + else: + with pytest.raises(SemVerError, match="Invalid semver constraint"): + parse_constraint(constraint_str) + # --- version_satisfies --- @pytest.mark.parametrize( diff --git a/tests/unit/test_versions.py b/tests/unit/test_versions.py new file mode 100644 index 0000000..afe1003 --- /dev/null +++ b/tests/unit/test_versions.py @@ -0,0 +1,43 @@ +import pytest + +from mthds.package.manifest.schema import MTHDS_STANDARD_VERSION, is_mthds_version_satisfied, is_valid_semver, is_valid_version_constraint +from mthds.package.semver import SemVerError +from mthds.protocol.protocol import PROTOCOL_VERSION + + +class TestVersions: + """The two numbers this library copies from the standard's cut, and how a manifest constraint is evaluated against one of them.""" + + def test_constants_match_the_standards_cut(self): + # Pinned deliberately: these are copies of a cut made in the standard + # (https://mthds.ai/latest/spec/versioning/), so a value moving here without a + # cut moving there is exactly what this asserts against. + assert MTHDS_STANDARD_VERSION == "2.0.0" + assert PROTOCOL_VERSION == "0.6.0" + assert is_valid_semver(MTHDS_STANDARD_VERSION) + assert is_valid_semver(PROTOCOL_VERSION) + + @pytest.mark.parametrize( + ("topic", "constraint", "is_satisfied"), + [ + ("the constraint every manifest written before the cut carries", ">=1.0.0", True), + ("an exact pin on the current standard", "2.0.0", True), + ("a caret range over the current major", "^2.0.0", True), + ("a tilde range over the current minor", "~2.0.0", True), + ("a wildcard over the current major", "2.*", True), + ("any standard version at all", "*", True), + ("a compound range spelled with a space, as the manifest format documents it", ">=2.0.0, <3.0.0", True), + ("the same compound range spelled without one", ">=2.0.0,<3.0.0", True), + ("a caret range over the superseded major", "^1.0.0", False), + ("a package that predates the cut and says so", "<2.0.0", False), + ("a package needing a standard that does not exist yet", ">=3.0.0", False), + ("a compound range closing below the current standard", ">=1.0.0, <2.0.0", False), + ], + ) + def test_manifest_constraint_evaluated_against_the_standard_version(self, topic: str, constraint: str, is_satisfied: bool): + assert is_valid_version_constraint(constraint), f"{topic}: the manifest format must accept this spelling" + assert is_mthds_version_satisfied(constraint) is is_satisfied, topic + + def test_unparsable_constraint_raises(self): + with pytest.raises(SemVerError, match="Invalid semver constraint"): + is_mthds_version_satisfied("not valid!!") From c3626649f8dac1fc82bfd5afc861cd684028fd59 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Tue, 8 Sep 2026 00:11:08 +0200 Subject: [PATCH 07/11] docs: cut the /release skill down to mthds-python's specifics (#66) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill now points at the workspace release play for the procedure and declares only what is this repo's own: what publish.yml ships to PyPI on the push to main and how the landing verifies it, the pyproject.toml version and the uv.lock that follows it, the gates including the protocol fixture corpus parity with mthds-js, the files the release commit carries, the workflows that gate a release pull request, and the particulars — no pre-release form can reach main, the changelog heading carries the v, and the back-merge is a merge commit. Everything the play does once for every repo is gone: the git status pre-flight that offered to fold uncommitted changes or unpushed commits into the release, the release branch created from the current HEAD in place, the numbered restatement of the bump, changelog, commit, push and pull request steps, and the post-merge reminder. The branch is now cut in _mthds-python--release by wt add, and the merge is landed by /ledger-land. Claude-Session: https://claude.ai/code/session_016F72qvy4QBZHe7XX24QmcT Co-authored-by: Claude Opus 5 --- .claude/skills/release/SKILL.md | 210 +++++++++----------------------- 1 file changed, 55 insertions(+), 155 deletions(-) diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index d0497a0..e47ccff 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -1,183 +1,83 @@ --- name: release description: > - Automates the mthds-python release workflow: bumps the version in - pyproject.toml, finalizes the CHANGELOG.md Unreleased section, runs quality - checks, regenerates uv.lock, creates a release/vX.Y.Z branch, commits, pushes, - and opens a PR to main. Use when user says "release", "cut a release", "bump - version", "prepare a release", "make a release", "ship it", "create release - branch", or any variation of shipping a new version of the mthds Python - package. The user can optionally provide changelog content inline when invoking - the skill (e.g. "/release Added a new base runner option"), which will be used - as the changelog entry for this version. + Cut a release of mthds-python, the `mthds` Python package published to PyPI: + the release/vX.Y.Z worktree, the pyproject.toml bump and the uv.lock that + follows, the changelog entry, the quality gates, the corpus parity check + against mthds-js, one commit, and a pull request to main. Use when the user + says "release", "cut a release", "bump version", "prepare a release", "make a + release", "ship it", "create release branch", "promote dev to main", "tag a + version", or any variation of shipping a new version of the mthds Python + package. Changelog content passed inline ("/release Added a base runner + option") becomes the entry. The merge is landed by /ledger-land, never by this + skill. --- -# mthds-python Release Workflow +# Releasing mthds-python -This skill handles the full release cycle for the `mthds` Python package (the -`mthds-python` repo). A release is a `release/vX.Y.Z` branch that PRs into -`main`; merging to `main` triggers `publish.yml`, which builds the wheel, -publishes it to PyPI as `mthds`, and creates the GitHub release from the -changelog notes. +The procedure is the workspace release play, [`docs/releasing.md`](../../../../docs/releasing.md) at the workspace root — read it first, then run it with what follows. The repo key is `mthds-python`, the base is `dev`, and the pull request targets `main`: `guard-branches.yml`'s `gate-main` job refuses any head branch but `release/vX.Y.Z` into `main`, so there is no other way in. The release worktree is `_mthds-python--release`, made with `wt add mthds-python release --branch release/vX.Y.Z`. The repo declares neither `.worktree.toml` nor `.worktreeinclude`, so `wt` resolves the base from `origin/dev` and provisions with the Makefile's `install` target, which is what creates the `.venv` every gate below runs out of. -## Files touched +## What ships -- **`pyproject.toml`** — the `version` field (line 3) -- **`CHANGELOG.md`** — add `[vX.Y.Z] - YYYY-MM-DD` entry (remove `[Unreleased]` if present) -- **`uv.lock`** — regenerated via `make li` (lock + install) +The merge to `main` publishes, from `publish.yml`, which fires on the push (`on: push: branches: [main]`) — so the run is keyed to `main` and its `headSha` is the merge commit itself: -## Workflow +- **The `mthds` package on PyPI.** The `build` job runs `python3 -m build`, and the `publish-to-pypi` job uploads the dists by trusted publishing into the `pypi` environment, . The upload passes no `skip-existing`, so a push to `main` carrying a version PyPI already holds fails there — and nothing else stands between an unbumped push and that failure, since `version-check.yml` runs on pull requests only. +- **The GitHub Release and the `vX.Y.Z` tag**, by the same workflow's `github-release` job, which needs both of the others. The notes are the changelog section for the version read out of `pyproject.toml` — everything from its `## [vX.Y.Z] - ` heading down to **two** lines above the next `## [v…] - ` heading, with blank lines dropped and each line's leading and trailing whitespace stripped — handed to `gh release create` alongside `--generate-notes`. The slice stops two lines short rather than one because the workflow subtracts a line twice, so the line sitting directly above the next heading never reaches the notes: harmless while that line is the blank separator the changelog format puts there, and a silently missing last bullet if an entry is ever written flush against the heading below it. Keep the blank line between sections. When no such heading is found the step warns, leaves the notes empty and exits 0, so the Release ships carrying the bare line `Release vX.Y.Z` instead of failing. The tag exists only as a side effect of `gh release create`; nothing in the workflow runs `git tag -a`, so always pass `--tags` when reading the tags with `git describe`. +- **Nothing else.** No documentation site is deployed: `docs/` ships inside the repo and no workflow publishes it anywhere. -### 1. Pre-flight checks +Sigstore signing runs **before** the Release is created and carries no `continue-on-error`, so a signing outage leaves the version published on PyPI with no tag and no GitHub Release at all. The cure is to re-run that run's **failed jobs**, which keeps the successful upload; a fresh full run would fail at the upload over the files already on PyPI. No workflow in this repo declares `workflow_dispatch`, so the run the push to `main` created is the only run there is. -- Read the current version from `pyproject.toml`. -- Read `CHANGELOG.md` to understand the current state. -- Run `git status` and `git log origin/main..HEAD` to assess the working tree: - - If there are **uncommitted changes** (staged or unstaged), warn the user and - ask whether to commit them as part of the release, stash them, or abort. - - If there are **unpushed commits** on the current branch, list them so the - user is aware — these will be included in the release branch. +The landing verifies the publish — the run, the registry's answer, the tag: -### 2. Determine the bump type - -Ask the user which kind of version bump they want — **patch**, **minor**, or -**major** — unless they already specified it. Show the current version and what -the new version would be for each option so the choice is concrete. - -While the package is pre-1.0 (`0.y.z`), treat the `0.MINOR.PATCH` segments the -way the project has been using them: a breaking change bumps the minor, a -backward-compatible feature or fix bumps the patch. If the changelog for this -release contains a `### Breaking Changes` section, steer the user toward at least -a minor bump. - -### 3. Run quality checks - -Run `make agent-check`. This is the gate — if it fails, stop and report the -errors so they can be fixed before retrying. Do not proceed past this step on -failure. - -### 4. Ensure we're on the right branch - -The release branch must be named `release/vX.Y.Z` where X.Y.Z is the **new** -version — `guard-branches.yml` rejects any other source branch merging into -`main`, and `version-check.yml` rejects a mismatch between the branch name and -the `pyproject.toml` version. All file modifications (changelog, version bump, -lock) must happen on this branch. - -- If already on `release/vX.Y.Z` matching the new version, stay on it. -- If on `dev`, `main`, or any other branch, create and switch to - `release/vX.Y.Z` from the current HEAD. -- If on a `release/` branch for a **different** version, warn the user and ask - how to proceed. - -### 5. Finalize the changelog - -Add a new version entry at the top of the changelog for the release. - -1. If there is an `## [Unreleased]` section, **remove it** (including any blank - lines that follow it) and replace it with the new version heading. Any - content that was under `[Unreleased]` becomes the content of the new version. -2. If there is no `[Unreleased]` section, insert the new version heading - directly after the `# Changelog` title. -3. **Never add an `[Unreleased]` heading.** The changelog should only contain - concrete version entries. -4. If the user provided changelog content when invoking the skill (e.g. - `/release Added a new base runner option`), **merge** that content with any - existing `[Unreleased]` content (do not discard either source). Format the - combined content properly under the appropriate headings — this repo uses - `### Breaking Changes`, `### Added`, `### Changed`, `### Fixed`, `### Removed` - — inferring headings from the content when possible. -5. If the release has no changelog content yet (neither from an `[Unreleased]` - section nor from inline user input), ask the user what to include before - proceeding. -6. The result should look like: - -```markdown -# Changelog - -## [vX.Y.Z] - YYYY-MM-DD - -### Changed -- ... - -## [vPREVIOUS] - PREVIOUS-DATE -... +```bash +gh run list --workflow=publish.yml --branch main --limit 3 --json conclusion,headSha,url # the run whose headSha is the merge commit: success +curl -s https://pypi.org/pypi/mthds/json | jq -r .info.version # the registry's answer: X.Y.Z +git fetch --tags --prune origin && git tag --list vX.Y.Z # the tag ``` -### 6. Bump the version in pyproject.toml - -Edit `pyproject.toml` line 3 to the new version string. Only change the version -field — don't touch anything else. +`gh release view vX.Y.Z` confirms the Release and its notes. The registry can equally be read with `pip index versions mthds` where a `pip` is on the PATH — the worktree's uv-made venv has none. -### 7. Lock dependencies +## Version files and the lock -Run `make li` to regenerate `uv.lock` and reinstall. This ensures the lockfile -reflects the new version in `pyproject.toml`. The `package-check.yml` CI job runs -`uv lock --locked` and fails the PR if `uv.lock` is out of sync, so this step is -not optional. If it fails, stop and report the error. +- **`pyproject.toml`** — the `[project]` table's `version`, and nothing else in the file. Keep it the file's **first** `version = ` line and keep it starting at column zero: `changelog-check.yml` and `publish.yml` both read it with `grep -m 1 'version = '`, and `version-check.yml` with `grep '^version'`. +- **`uv.lock`** — regenerated by `make li` (`make lock` then `make install`, that is `uv lock` followed by `uv sync --all-extras`), run after the bump so the lockfile records the new number. If it fails, stop and report it rather than committing a stale lock: `package-check.yml` runs `uv lock --locked` and diffs the file on every pull request. +- **Also stamped: nothing.** The release number lives in `pyproject.toml` alone — no `__version__`, no README badge, no literal in `docs/`. The other version numbers the package declares are **not** this one, and a release never touches them: `MTHDS_STANDARD_VERSION` in `mthds/package/manifest/schema.py` and `PROTOCOL_VERSION` in `mthds/protocol/protocol.py` are copies of cuts made by the MTHDS standard, moved when the standard moves and never because this library is shipping. `docs/versioning.md` is the account of every number in play and which of them this repo owns. -### 8. Commit and push +## Gates -Stage all release-related changes. This includes at minimum `pyproject.toml`, -`CHANGELOG.md`, and `uv.lock`, plus any other files the user chose to include in -step 1 (e.g. previously uncommitted work that belongs in this release). +Run in the worktree, in this order, before the commit: -Commit with the message: +1. **`make agent-check`** — unused imports, ruff format, ruff lint, pyright, mypy. **It rewrites files** (`fix-unused-imports`, `format`, and `lint`, which is `ruff check --fix`), so whatever it touches joins the release commit. These are the fixing counterparts of exactly what `lint-check.yml` runs read-only on the pull request — `merge-check-ruff-format`, `merge-check-ruff-lint`, `merge-check-pyright`, `merge-check-mypy` — across every supported Python, so a red one here is a red pull request there. Red blocks the release: fix the code, never loosen the target. `make check` additionally runs pylint, which no workflow does; it is not part of this flow. +2. **`make agent-test`** — the pytest suite, quiet unless it fails. `tests-check.yml` runs `make gha-tests` (`pytest --exitfirst --quiet`) on every supported Python, so a failure here is the pull request's failure too. +3. **The protocol fixture corpus, against `mthds-js`.** `tests/fixtures/protocol/` is committed byte-identically in this repo and in `mthds-js`, and `protocol-corpus-parity.yml` checks the sibling out on every pull request and fails on any difference, so a release promoting a one-sided recapture opens red. Read the sibling's checkout before committing, with the workflow's own exclusions: -``` -Release vX.Y.Z -``` - -Push the branch to origin with `-u` to set up tracking. + ```bash + diff --recursive --brief -x README.md -x '*.fixture.ts' tests/fixtures/protocol ../mthds-js/tests/fixtures/protocol + ``` -### 9. Open a PR + That reads `mthds-js`'s main checkout, which under the main-checkout invariant sits on `dev` — the branch CI falls back to — so bring it current before trusting a green result, and treat the CI job as the authority. A difference is never repaired on the release branch: the capture is regenerated from `pipelex` and copied into both mirrors at once, which is the `projection-corpus-update` skill's job and a separate pull request in each repo, and the release waits for both. The mechanics are `tests/fixtures/protocol/README.md`, and the cross-repo authority is `conformance/scripts/check-protocol-fixture-parity.py`. -Create a pull request targeting `main` with: +## The release commit -- **Title:** `Release vX.Y.Z` -- **Body:** Include: - - The changelog entries for this version (copied from CHANGELOG.md) - - A note about the version bump from old to new +`pyproject.toml`, `uv.lock`, `CHANGELOG.md`, and each file `make agent-check` rewrote — staged by name. -Use this format for the PR body: +## CI on the release pull request -```markdown -## Release vX.Y.Z +- **`guard-branches.yml`** (`gate-main`) — the head branch into `main` matches `^release/v[0-9]+\.[0-9]+\.[0-9]+$` exactly. Its `protect-workflows` job additionally refuses a change under `.github/workflows/` from an author whose effective repository permission is not `write` or `admin`. +- **`version-check.yml`** — the `pyproject.toml` version equals the version in the branch name. A head that does not match the release form does not slip past it: the first step calls a non-match "skipping" and exits 0, but that ends the step alone, and the comparison in the last step then runs against an empty branch version and fails. +- **`changelog-check.yml`** — `CHANGELOG.md` carries a `## [vX.Y.Z] - ` heading for the version in `pyproject.toml`. It asserts nothing about `[Unreleased]`; leaving none behind is the play's rule, not CI's. +- **`package-check.yml`** (`uv-lock-check`, on every pull request) — `uv lock --locked` leaves `uv.lock` unchanged. +- **`lint-check.yml`** — the merge checks on every supported Python; the aggregator job `Lint (all versions)` is the single required status. +- **`tests-check.yml`** — `make gha-tests` on every supported Python; the aggregator job `Tests (all)` is the single required status. +- **`protocol-corpus-parity.yml`** — `tests/fixtures/protocol/` byte-identical with `mthds-js`, excluding `README.md` and the `*.fixture.ts` twins. It prefers the sibling's branch of the same name as the head and falls back to the sibling's `dev`. A `release/vX.Y.Z` head therefore reaches `dev` only when `mthds-js` holds no branch of that exact name: a release branch left on the sibling's origin after its own merge would be matched instead, and the two repos' version lines are independent but not disjoint. The job's `Resolve the sibling ref` step prints the ref it chose — read that rather than assuming `dev`. +- **`cla.yml`** — the CLA assistant, allowlisted through the `CLA_ALLOWLIST` repository variable. -Bumps version from `A.B.C` to `X.Y.Z`. +Nothing in CI compares the new version against the one already on `main`: `version-check.yml` checks equality with the branch name and no more, so a version that is not an increase passes the pull request and fails after the merge, at the PyPI upload. -### Changelog - - -``` +## Particulars -Report the PR URL back to the user, and remind them that **merging the PR into -`main` is what publishes** — `publish.yml` builds and pushes the package to PyPI -and cuts the GitHub release automatically. Nothing publishes until the PR is -merged. - -## Important details - -- The version follows semver: `MAJOR.MINOR.PATCH`. -- Always confirm the bump type with the user before making changes. -- If `make agent-check` fails, the release is blocked — help the user fix the - issues rather than skipping the checks. -- The CI gates a `release/vX.Y.Z` → `main` PR with: - - `version-check.yml` — the `pyproject.toml` version must match the - `release/vX.Y.Z` branch name. - - `changelog-check.yml` — `CHANGELOG.md` must contain a `## [vX.Y.Z] -` entry - for the new version. - - `package-check.yml` — `uv.lock` must be in sync with `pyproject.toml` - (`uv lock --locked`). - - `tests-check.yml` — the test matrix must pass on every supported Python - version (3.10 through 3.14). - - `lint-check.yml` — formatting, lint, and type checks (the same gates as - `make agent-check`). - - `guard-branches.yml` — only `release/vX.Y.Z` branches may target `main`. -- All checks must pass for the PR to be mergeable, so getting the changelog, - version, and lockfile right is critical. -- Pre-release versions are supported: a PEP 440 suffix (`a`, `b`, or `rc` - followed by a number, e.g. `0.6.0rc1`) makes `publish.yml` mark the GitHub - release as a pre-release. Use the branch name `release/v0.6.0rc1` to match. -- Today's date for the changelog entry: use the current date in `YYYY-MM-DD` - format. +- **No pre-release form reaches `main`.** `publish.yml` does detect a PEP 440 `a`, `b` or `rc` suffix and marks the GitHub Release a pre-release, which reads like permission — but nothing can carry such a version through: `guard-branches.yml` refuses a `release/v0.6.0rc1` head into `main` outright, and `version-check.yml` fails on the mismatch between a suffixed `pyproject.toml` version and the plain number in a `release/vX.Y.Z` branch name. Ship a plain `X.Y.Z`. The `pre-release/v*` base branch that `guard-branches.yml`'s `gate-release` job tolerates publishes nothing either, since `publish.yml` fires only on the push to `main`. +- **The changelog heading carries the `v`** — `## [vX.Y.Z] - YYYY-MM-DD`, which is exactly what `changelog-check.yml` greps for and what `publish.yml` slices the Release notes out of. No `[Unreleased]` heading is left behind; the next change re-creates one. +- **The fixture corpus is half of a pair.** A capture lands in `mthds-python` and `mthds-js` or in neither, so a release is also a statement that the sibling's `dev` carries the same bytes. That is what gate 3 and `protocol-corpus-parity.yml` are for. +- **A release arms no follow-ups on its own.** The ledger's `ledger.toml` declares no `release_followups` for this repo, so anything that must happen after this version ships is filed by hand alongside the release item in the play's step 3. +- **The back-merge is a merge commit.** `dev` carries a `Merge main into dev after the vX.Y.Z release` commit after each release rather than a fast-forward; `/ledger-land` makes it, and `CHANGELOG.md` is the one conflict it expects. From 7cd4a37854db3f0309d468c4348b18d2bfe06b33 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sat, 12 Sep 2026 19:50:13 +0200 Subject: [PATCH 08/11] =?UTF-8?q?feature/Method-file-serialization=20?= =?UTF-8?q?=C2=B7=20L-260907-c34629=20=E2=80=94=20blank=20is=20ECMAScript-?= =?UTF-8?q?blank=20(#67)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `mthds.protocol.method_files` — `MethodFile` and the `serialize_method_files` / `parse_method_files` pair — the Python twin of the `mthds` npm package's `protocol/method_files.ts`, so a Python client reads a stored method's catalog source (`[{ name, content }]`, with `""` as the no-source form) without re-porting the platform's decoder. The port is held to the twin edge for edge: the blank predicate is ECMAScript `trim`'s closed set rather than `str.strip()`, which disagrees at both ends, and the bytes are `JSON.stringify`'s; the suite pins both against output the twin printed, and the docstring names the three places the twin deliberately departs from the platform. Closes L-260907-c34629 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 --- CHANGELOG.md | 6 + README.md | 2 +- docs/runners.md | 27 ++- mthds/protocol/method_files.py | 278 +++++++++++++++++++++++ pyproject.toml | 8 + tests/unit/test_config_dialect.py | 2 +- tests/unit/test_data.py | 131 +++++++++++ tests/unit/test_protocol_method_files.py | 193 ++++++++++++++++ 8 files changed, 644 insertions(+), 3 deletions(-) create mode 100644 mthds/protocol/method_files.py create mode 100644 tests/unit/test_protocol_method_files.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 97836cc..ea5a123 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## [Unreleased] + +### Added + +- **`mthds.protocol.method_files` — the catalog serialization of a stored method's source**: `MethodFile` (`name` + `content`) and the pair `serialize_method_files` / `parse_method_files` between a list of files and the JSON `[{ name, content }]` string the hosted platform persists for a method's `.mthds` source and its custom PipeFunc `python`. The empty list serializes to `""`, the platform's "no source" sentinel, never to `"[]"`; blank-content entries are dropped in both directions; anything but the named array raises `PipelineRequestError`, including the three constants Python's `json` accepts and `JSON.parse` refuses. It mirrors the `mthds` npm package's `protocol/method_files.ts` edge for edge — blank is ECMAScript-blank, and the bytes are `JSON.stringify`'s down to both surrogate rules, an unpaired one escaped and an adjacent pair combined into the astral character Python alone can spell two ways — so a Python client reads a stored method's bundle from here rather than re-porting the platform's decoder. Python's own number model is kept out of the way too: integers decode as floats, the twin's single number type, so a literal longer than `sys.get_int_max_str_digits()` cannot raise a bare `ValueError` through the typed surface. One divergence from the twin is forced rather than chosen and is documented as such: Python's decoder recurses where `JSON.parse` iterates, so a deeply nested source is refused here and accepted there, at a depth that is an interpreter build constant. + ## [v0.14.0] - 2026-09-06 ### Added diff --git a/README.md b/README.md index aaaa95f..4ad410f 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ pip install mthds ## What's in the box -- **The protocol** (`mthds.protocol`) — `MTHDSProtocol`, the five-route interface every runner implements (`execute`, `start`, `validate`, `models`, `version`); the wire models (`RunResult`, `ModelDeck`, `ValidationReport`, `VersionInfo`); and the domain shapes methods are built from (`concept`, `stuff`, `working_memory`, `pipe_output`, `pipeline_inputs`). +- **The protocol** (`mthds.protocol`) — `MTHDSProtocol`, the five-route interface every runner implements (`execute`, `start`, `validate`, `models`, `version`); the wire models (`RunResult`, `ModelDeck`, `ValidationReport`, `VersionInfo`); the domain shapes methods are built from (`concept`, `stuff`, `working_memory`, `pipe_output`, `pipeline_inputs`); and the catalog serialization of a stored method's source (`method_files`). - **Runners** (`mthds.runners`) — `MthdsAPIClient` (`mthds.runners.api.client`), the HTTP client for any MTHDS-Protocol server plus the hosted run-lifecycle (polling) extension; and `PipelexRunner` (`mthds.runners.pipelex.runner`), which shells out to a locally installed `pipelex` CLI. - **Package management** (`mthds.package`) — read, lock, and resolve `METHODS.toml` manifests. diff --git a/docs/runners.md b/docs/runners.md index 6d3aca1..d5ed60d 100644 --- a/docs/runners.md +++ b/docs/runners.md @@ -6,7 +6,7 @@ The `mthds` package is the Python client for the open-source `pipelex-api` runne The protocol contract and its implementations live in separate packages: -- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). +- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `method_files.py` (the catalog serialization of a stored method's source — see [The catalog serialization](#the-catalog-serialization-method_files) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). - `mthds/runners/` — every runner implementation, one subpackage per runner: - `api/` — the API runner: `client.py` (`MthdsAPIClient`, one file with its helpers), `models.py` (the Dict-serialized wire models — `DictConcept`, `DictStuffAbstract`, `DictWorkingMemoryAbstract`, `DictPipeOutputAbstract`, `DictRunResultExecute` — the runners' concrete JSON materialization of the protocol's domain shapes), `exceptions.py` (API auth + the protocol's 202-degrade error, `RunStillRunningError`). - `pipelex/runner.py` — `PipelexRunner`, the local runner that shells out to the `pipelex` CLI. @@ -110,6 +110,31 @@ What to know about the two modules: - **Parity with the TypeScript client is measured.** `tests/fixtures/protocol/` holds one real payload pair from the reference engine, committed byte-for-byte here and in `mthds-js`; the suite parses it strictly and asserts the dump equals the input. The fixture's README records the capture and the known engine drift the standard has since ruled on. - **The descriptor projects to a fill-in inputs template, client-side.** `mthds.protocol.inputs_template` turns one `PipeInputFormDescriptor` into the template somebody fills in and hands back — the compact and explicit shapes, as JSON or as TOML — so a client needs no server round trip to offer one for a method it does not have on disk. It is held to byte identity with the twin projection in the `mthds` npm package. See [docs/inputs-template.md](./inputs-template.md). +### The catalog serialization: `method_files` + +A stored method's source is persisted by the hosted platform as one string in the **catalog** form — the JSON `[{ name, content }]` array the webapp editor writes, one entry per bundle-relative file, for the method's `.mthds` source and again for its custom PipeFunc `python`. `mthds.protocol.method_files` is the one Python definition of that form: `MethodFile` (`name` + `content`, a closed strict shape) and the pair `serialize_method_files` / `parse_method_files` between a list of them and the stored string. It mirrors `protocol/method_files.ts` in the `mthds` npm package, so a client that fetches a stored method and wants its bundle back — an MCP server, your own code — reads it here instead of re-porting the platform's decoder. That is what the module is for, not yet a description of every consumer: `pipelex-sdk` still carries its own copy of this format, which disagrees with this one on blankness and on bytes, and moving it across is tracked as ledger item L-260912-6ef508. + +```python +from mthds.protocol.method_files import MethodFile, parse_method_files, serialize_method_files + +files = parse_method_files(stored_method["mthds"]) # -> list[MethodFile], in stored order; "" and "[]" both give [] +for method_file in files: + print(method_file.name, len(method_file.content)) + +stored = serialize_method_files([MethodFile(name="bundle.mthds", content=bundle_text)]) # -> the catalog string +``` + +What to know about it: + +- **It is the at-rest representation, not the run surface.** The catalog is an ordered, named array; a run request's `files` is an unordered path-to-text map. They are different shapes for different moments, and this module owns only the first. +- **No files is `""`, never `"[]"`.** The empty string is the platform's "no source" / "clear the field" sentinel, so the empty list serializes to it; the literal `"[]"` would be stored as a source. On the way back, a blank string, `None` and the empty array all parse to `[]`. +- **A blank file is not a file.** An entry whose content is empty or whitespace-only is dropped on both directions, so the round-trip is stable and the canonical form never carries an empty file. "Whitespace" here is what ECMAScript's `trim` strips — the byte-order mark included, U+001C–U+001F and U+0085 excluded — because the twin's verdict on a file must be this side's, and Python's `str.isspace` disagrees at both ends. One test derives that set from the Unicode categories rather than sampling it, so a code point cannot quietly go missing. Note that a zero-byte file is therefore not representable: an empty `__init__.py` does not survive a round trip. +- **The bytes are the twin's.** Serialization writes what `JSON.stringify` writes: compact separators, UTF-8 left as is, and only the JSON-mandated escapes. Surrogates take two rules, because Python addresses code points where JavaScript addresses UTF-16 code units: an unpaired surrogate is escaped rather than written raw — `ensure_ascii=False` alone would produce a string that cannot be encoded as UTF-8 at all — while two adjacent halves of a pair, which Python can hold as two code points and the twin cannot, are combined into the astral character the twin writes. The suite pins both against strings the twin printed. +- **Anything but the named array is a contract violation.** A source that is not JSON — a legacy raw bundle string included — a non-array value, an entry that is not an object with string `name` and string `content`, one of the three constants Python's `json` accepts and `JSON.parse` refuses (`NaN`, `Infinity`, `-Infinity`, an ignored extra member included) all raise `PipelineRequestError`. An entry's extra members are tolerated and stripped; a blank name is not a reason to drop an entry; duplicate names are kept, and since the platform refuses a method whose file names collide, uniqueness is the producer's obligation. An oversized integer is *not* a violation: Python's bounded `int()` conversion would refuse bytes the twin accepts, so integers are decoded as floats, which is the twin's own number model. +- **Deep nesting is the one place the two sides differ, and not by choice.** A source nested deeply enough to exhaust Python's recursive decoder raises `PipelineRequestError` rather than letting a `RecursionError` escape the typed surface — but the twin's `JSON.parse` is iterative and accepts any depth, so this side refuses what the twin accepts. The depth that trips it is an interpreter build constant: it rose by an order of magnitude in CPython 3.14 and differs between patch releases, so treat the guard as best-effort and never as a depth you can rely on. +- **A `name` is untrusted text.** Neither this module nor the twin checks that it is really bundle-relative, so an absolute or `..`-walking name round-trips untouched. Validate containment before writing a catalog to disk. +- **Where it departs from the platform's own decoder, it follows the twin.** The platform has no blank-source guard on its contents path at all, so a whitespace-only raw source fails downstream at the TOML parse; its own blankness is Python's `str.strip()`, which disagrees with the twin in both directions on an entry's content (a BOM-only file is content there and blank here, a U+001C–U+001F or U+0085 file the reverse); it drops blank-named entries; and it reads a non-array source — or a whole array containing one malformed entry — as a single legacy bundle, while silently dropping an entry whose content is not a string. This module and the TypeScript twin agree with each other on every one of those instead, so the three implementations never split two against one. **This package does not decode the legacy bare-bundle shape**, so a consumer reading one of the platform's very old plain-string rows must handle the `PipelineRequestError` itself — the example above assumes a catalog-form row. + ### Basic args vs extension args The abstract `MTHDSProtocol` interface carries the protocol's **basic** arguments only. Implementations may accept more (the protocol's extension policy), and the SDK passes any of them through: diff --git a/mthds/protocol/method_files.py b/mthds/protocol/method_files.py new file mode 100644 index 0000000..306572f --- /dev/null +++ b/mthds/protocol/method_files.py @@ -0,0 +1,278 @@ +"""The canonical **catalog serialization** of a method's source files — mirrors `mthds-js`'s `protocol/method_files.ts`. + + catalog string ──parse_method_files──▶ list[MethodFile] (ordered, named) + list[MethodFile] ──serialize_method_files──▶ catalog string + +The catalog string is the JSON `[{ name, content }]` array form (the webapp-editor format) +that the hosted platform persists for a stored method's `.mthds` source and its custom +PipeFunc `python`. One canonical type plus one (de)serializer, so every Python consumer — +`pipelex-sdk`, an MCP server, a client that fetches a stored method and wants its bundle +back — agrees on the shape instead of each re-porting the platform's decoder and drifting. +The TypeScript twin exists for the same reason on its side; the two are held to the same +behaviour, edge for edge, and the test suite pins the Python side against bytes produced by +running the twin. + +This is the AT-REST catalog representation and is distinct from the run-surface `files` +map (an unordered path-to-text mapping on a run request): the catalog stores an ordered, +named ARRAY. The empty list is serialized as the empty string `""` — the platform's "no +source" / "clear the field" sentinel — never as the literal `"[]"`. + +The rules a port gets wrong while still round-tripping the happy case, all stated here: + +- **Blank means ECMAScript-blank.** A file carries no source when its content is empty or + whitespace-only, and "whitespace" is what `String.prototype.trim` strips — the closed set + in `_ECMASCRIPT_TRIM_CHARS` below — not what Python's `str.isspace` says. The two sets + disagree at both ends: Python strips U+001C to U+001F and U+0085, which the twin keeps as + content; the twin strips the byte-order mark U+FEFF, which Python keeps. A file the twin + drops must be a file this side drops, so the predicate is spelled out rather than borrowed + from `str.strip()`, and a test derives it from the Unicode categories rather than sampling it. +- **The bytes are `JSON.stringify`'s.** Compact separators, non-ASCII text left as UTF-8, and + only the characters JSON requires escaped. Python's `json.dumps` defaults to spaced + separators and backslash-u escapes for anything outside ASCII: same value, different bytes, + and an at-rest representation that hashes or compares differently on the two sides. +- **Surrogates are where Python's code points and the twin's UTF-16 code units disagree.** A + Python `str` addresses code points, so it can hold a lone surrogate *and* it can hold the two + halves of a pair as two adjacent code points; a JavaScript string addresses code units, so that + same sequence is one astral character to the twin. `ensure_ascii=False` writes either raw, and + for a lone surrogate the result cannot be encoded as UTF-8 at all — a break at the storage + boundary rather than at this typed surface. The twin's well-formed `JSON.stringify` escapes a + lone surrogate and emits a pair as the character it encodes, so `_coalesce_surrogate_pairs` + combines pairs before the dump and `_escape_lone_surrogates` escapes whatever is left after it. +- **`json.loads` accepts three constants `JSON.parse` refuses.** `NaN`, `Infinity` and + `-Infinity` are a Python extension to JSON. A catalog string carrying one anywhere parses here + and throws in the twin — and an ignored extra member is the case that bites, because no later + shape check ever looks there. `parse_constant` refuses all three, so the two sides do not + return opposite verdicts on those bytes. +- **Python's integer conversion is bounded and the twin's is not.** `json.loads` runs every + integer through `int()`, which since 3.11 refuses a literal longer than + `sys.get_int_max_str_digits()` with a bare `ValueError` — not a `JSONDecodeError` — so a long + enough integer inside an ignored extra member escaped this typed surface entirely while the twin + parsed the same bytes and stripped the member. JavaScript has one number type and no such limit, + so integers are decoded as floats here (`parse_int=float`): that is the twin's own model, it is + immune to the limit, and nothing kept is affected, because every number on this surface is + either discarded with its extra member or refused by the shape check. + +One divergence from the twin is forced rather than chosen, and it is the only one: `JSON.parse` is +iterative and accepts nesting of any depth, while Python's decoder recurses and raises +`RecursionError` past a threshold that is an interpreter build constant — it moved by an order of +magnitude in CPython 3.14, and it differs even between patch releases. That error is caught and +re-raised as a `PipelineRequestError` rather than left to escape this typed surface, so a source +nested deeply enough is refused here and accepted there. The guard is best-effort by nature: +the depth that trips it cannot be predicted, so the test discovers it rather than pinning it. + +Where this deliberately departs from the platform's own decoder (`method_source_to_contents` +and `parse_stored_files` in `pipelex-server`'s platform member), it follows the twin, so that +the three implementations do not split two against one: + +- The platform has no blank-source guard on the contents path at all — `method_source_to_contents` + goes straight to `json.loads` and hands back the raw string when that fails — so a + whitespace-only raw source becomes a whitespace bundle there and fails downstream at the TOML + parse; here, as in the twin, it is "no source". +- The platform's own blankness is Python's `str.strip()`, so it disagrees with the twin in both + directions on an entry's content: a BOM-only file is content to the platform and blank here, + while a file of U+001C to U+001F or U+0085 is blank to the platform and content here. Here, as + in the twin, blankness is ECMAScript's. +- The platform drops an entry whose `name` is blank; here, as in the twin, the name is not + inspected — only a blank `content` drops an entry. +- The platform treats a source that is not the file-array form as one legacy bare bundle, reads + a whole array as one such bundle as soon as a single entry is malformed, and drops an entry + whose `content` is not a string while keeping the rest. Here, as in the twin, every one of + those is a contract violation of this typed surface, and **this package does not decode the + legacy bare-bundle shape at all**: a consumer holding one of the platform's very old + plain-string rows must handle the `PipelineRequestError` itself, because the catalog + file-array is always the named-array form. + +Two obligations this surface does not discharge, which its consumers therefore own: + +- **A `name` is untrusted text, not a checked path.** Neither this module nor the twin validates + it, so `../../../etc/passwd`, an absolute path, a Windows path and the empty string all + round-trip untouched. Anything that materializes a catalog to disk must check containment + itself; this module only preserves what the catalog held. +- **Duplicate names are kept here and refused downstream.** The catalog form permits them, and + the platform rejects a method whose `.mthds` and `.py` file names collide, so uniqueness is + the producer's obligation rather than something a parse will tell it about. +""" + +from __future__ import annotations + +import json +import re +from typing import TYPE_CHECKING, Any, NoReturn, cast + +from pydantic import BaseModel, ConfigDict + +from mthds.protocol.exceptions import PipelineRequestError + +if TYPE_CHECKING: + from collections.abc import Sequence + + +class MethodFile(BaseModel): + """One named source file of a method bundle: a bundle-relative path and its text. + + A closed, strict shape — a consumer building one gets exactly `name` and `content`, both + strings. Leniency toward what the wire carries (extra members, which are tolerated and + stripped) belongs to `parse_method_files`, not to the type. + """ + + model_config = ConfigDict(extra="forbid", strict=True) + + name: str + """Bundle-relative path, e.g. `"funcs/price.py"` or `"bundle.mthds"`. + + Untrusted text, preserved exactly as the catalog held it: neither this type nor the twin + checks that it is in fact relative, so a consumer writing it to disk validates containment. + """ + + content: str + """The file's UTF-8 text content.""" + + +# The code points ECMAScript's `String.prototype.trim` strips — WhiteSpace (TAB, VT, FF, SP, +# NBSP, ZWNBSP and every Zs-category character) plus the four LineTerminators — spelled out +# so the blank predicate is the twin's, not `str.isspace`'s. Python's set is wider on the +# control side (it strips U+001C to U+001F and U+0085, which JavaScript keeps) and narrower at +# U+FEFF (which JavaScript strips); the tests pin every one of those disagreements, and one of +# them derives this set from the Unicode categories so a dropped code point cannot pass. +_ECMASCRIPT_TRIM_CHARS = ( + "\t\n\v\f\r \u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000\ufeff" +) + + +def _is_blank(text: str) -> bool: + """A file carries no source when its content is empty or ECMAScript-whitespace-only.""" + return text.strip(_ECMASCRIPT_TRIM_CHARS) == "" + + +# A Python `str` addresses code points, so it can hold the two halves of a UTF-16 surrogate pair +# as two adjacent code points. A JavaScript string addresses code units, so that same sequence is +# one astral character to the twin, whose well-formed `JSON.stringify` (ES2019) writes it raw and +# escapes only a surrogate that is genuinely unpaired. Combining the pairs first is therefore what +# makes the two sides agree on a value only Python can spell two ways. +_SURROGATE_PAIR = re.compile(r"([\ud800-\udbff])([\udc00-\udfff])") + +# Whatever survives that pass cannot be part of an astral character, and +# `json.dumps(ensure_ascii=False)` writes it raw, producing a `str` that cannot be encoded as +# UTF-8 at all — so the twin's escape is reapplied after the dump. +_LONE_SURROGATE = re.compile(r"[\ud800-\udfff]") + + +def _coalesce_surrogate_pairs(text: str) -> str: + """Combine each adjacent high+low surrogate into the astral character it encodes, as the twin holds it. + + Left to right, so a lone high surrogate immediately before a well-formed pair stays lone — + which is what the twin does with the same code units. + """ + + def combine(match: re.Match[str]) -> str: + high, low = match.group(1), match.group(2) + return chr(0x10000 + ((ord(high) - 0xD800) << 10) + (ord(low) - 0xDC00)) + + return _SURROGATE_PAIR.sub(combine, text) + + +def _escape_lone_surrogates(dumped: str) -> str: + """Escape unpaired surrogates the way `JSON.stringify` does, leaving every other character raw.""" + return _LONE_SURROGATE.sub(lambda match: f"\\u{ord(match.group()):04x}", dumped) + + +def _refuse_json_constant(constant: str) -> NoReturn: + """Refuse the three constants `json.loads` accepts as an extension and `JSON.parse` rejects.""" + msg = f"Method file source is not valid JSON; expected a [{{ name, content }}] array (got `{constant}`)." + raise PipelineRequestError(msg) + + +def serialize_method_files(files: Sequence[MethodFile]) -> str: + """Serialize method files to the canonical catalog string. + + Blank-content entries are dropped (a zero-source file is not persisted), so the canonical + form never carries an empty file — which also means a list of nothing but blank files and an + empty list are the same call, both signalling "clear the stored source". An empty result + serializes to `""` — the platform's "no source" / "clear the field" signal — not `"[]"`. The + bytes are those of the twin's `JSON.stringify`: compact, UTF-8 left as is, only the + JSON-mandated escapes, and an unpaired surrogate escaped rather than written raw. + + Args: + files: The files, in the order they are to be stored. + + Returns: + The catalog string, or `""` when no file carries source. + """ + kept: list[dict[str, str]] = [ + {"name": _coalesce_surrogate_pairs(method_file.name), "content": _coalesce_surrogate_pairs(method_file.content)} + for method_file in files + if not _is_blank(method_file.content) + ] + if not kept: + return "" + return _escape_lone_surrogates(json.dumps(kept, ensure_ascii=False, separators=(",", ":"))) + + +def parse_method_files(source: str | None) -> list[MethodFile]: + """Parse the canonical catalog string back into method files. + + A blank source (`""` / whitespace / `None`) and an empty JSON array both yield `[]`. A JSON + `[{ name, content }]` array yields those files in order, with blank-content entries dropped + (mirroring serialization, so the round-trip is stable) and any extra member of an entry + ignored. Duplicate names are kept, and a blank name is not a reason to drop an entry. + + Anything else — a non-array JSON value, an array entry that is not an object with string + `name` and string `content`, unparseable text (a legacy raw bundle string included), or one of + the three constants `json.loads` accepts as an extension and `JSON.parse` refuses (`NaN`, + `Infinity`, `-Infinity`) — is a contract violation for this typed surface and raises + `PipelineRequestError`. A source that exhausts the decoder raises it too, but as a best-effort + guard rather than a shared rule: the twin's `JSON.parse` is iterative and accepts nesting this + side refuses, which is the module's one forced divergence from it. + + Args: + source: The stored catalog string, or `None` for a field that was never set. + + Returns: + The files the catalog holds, in stored order. + """ + if source is None or _is_blank(source): + return [] + + parsed: Any + try: + # `parse_int=float` is the twin's own number model — JavaScript has one number type — and + # it sidesteps Python's bounded `int()` conversion, whose bare `ValueError` would otherwise + # escape this surface. `ValueError` below subsumes `JSONDecodeError` and covers any other + # decoder refusal; `RecursionError` is not a `ValueError`, so it stays named. + # `PipelineRequestError` does not subclass `ValueError`, so the constant refusal above + # passes through carrying its own message — if that base ever changes, this clause has to + # re-raise it before wrapping, or that message is lost. + parsed = json.loads(source, parse_constant=_refuse_json_constant, parse_int=float) + except (ValueError, RecursionError) as exc: + msg = "Method file source is not valid JSON; expected a [{ name, content }] array." + raise PipelineRequestError(msg) from exc + + if not isinstance(parsed, list): + msg = "Method file source must be a JSON array of { name, content } entries." + raise PipelineRequestError(msg) + + files: list[MethodFile] = [] + for entry in cast("list[Any]", parsed): + method_file = _method_file_from_entry(entry) + if method_file is None: + msg = "Each method file entry must be an object with string `name` and string `content`." + raise PipelineRequestError(msg) + if not _is_blank(method_file.content): + files.append(method_file) + return files + + +def _method_file_from_entry(entry: Any) -> MethodFile | None: + """Read one array entry as a file, or `None` when it is not the `{ name, content }` shape. + + Only the two members are read and both must be strings; anything else the entry carries + is ignored, as the twin ignores it. + """ + if not isinstance(entry, dict): + return None + members = cast("dict[str, Any]", entry) + name = members.get("name") + content = members.get("content") + if not isinstance(name, str) or not isinstance(content, str): + return None + return MethodFile(name=name, content=content) diff --git a/pyproject.toml b/pyproject.toml index 9194e6c..e30259f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -178,6 +178,13 @@ strictParameterNoneValue = true strictSetInference = true typeCheckingMode = "strict" +# Scoped once here rather than as a per-line ignore in each suite: a test may legitimately pin a +# module-private name whose exact value IS the contract, such as the ECMAScript trim set the +# catalog serialization's blank predicate is built from. Everything else stays strict. +[[tool.pyright.executionEnvironments]] +root = "tests" +reportPrivateUsage = "none" + [tool.ruff] exclude = [ ".cursor", @@ -326,6 +333,7 @@ convention = "google" "tests/**/*.py" = [ "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise + "import-private-name", # A test may pin a module-private constant whose exact value IS the contract, e.g. the ECMAScript trim set ] "examples/**/*.py" = [ "implicit-namespace-package", # Runnable demo scripts, not an importable package diff --git a/tests/unit/test_config_dialect.py b/tests/unit/test_config_dialect.py index 2dd9079..e57509b 100644 --- a/tests/unit/test_config_dialect.py +++ b/tests/unit/test_config_dialect.py @@ -14,7 +14,7 @@ # The dialect engine is deliberately private (an implementation detail of the config surface); # this suite pins the dialect itself, so it reaches the private names on purpose. -from mthds.config import _parse_dotenv, _serialize_dotenv # ruff: ignore[import-private-name] # pyright: ignore[reportPrivateUsage] +from mthds.config import _parse_dotenv, _serialize_dotenv # pyright: ignore[reportPrivateUsage] _FIXTURE_PATH = Path(__file__).resolve().parent.parent / "fixtures" / "config_dialect_cases.json" diff --git a/tests/unit/test_data.py b/tests/unit/test_data.py index 158c264..c69306d 100644 --- a/tests/unit/test_data.py +++ b/tests/unit/test_data.py @@ -590,3 +590,134 @@ class CompactSlotCases: item_count=2, item=ObjectItem(concept_ref="probe.Gadget", required=True, fields=[TextField(name="label", required=True)]), ) + + +class MethodFileCases: + """Vectors for the catalog serialization, pinned against the TypeScript twin. + + `TWIN_SERIALIZED` is the exact string `serializeMethodFiles(TWIN_FILES)` printed when run from + `mthds-js/src/protocol/method_files.ts` (2026-09-12, node 24): compact separators, UTF-8 left as + is, the double quote, the backslash and the LF and TAB it happens to contain escaped and nothing + else. `BLANK_PREDICATE` is what the same run said of each code point — which single-character + contents `serializeMethodFiles` dropped and which `parseMethodFiles` read as "no source" — so + every disagreement between ECMAScript's `trim` and Python's `str.isspace` is stated as a case + rather than assumed away. It samples the set; `test_the_trim_set_is_ecmascripts_whole_trim_set` + is what actually holds the set complete, by deriving it. + """ + + TWIN_FILES: ClassVar[list[dict[str, str]]] = [ + { + "name": "bundle.mthds", + "content": 'domain = "x"\n[concept.\u00c9t\u00e9]\n' + 'description = "caf\u00e9 \u2014 \u00fcn\u00efc\u00f6d\u00e9 \u65e5\u672c \U0001f642"\n', + }, + {"name": "funcs/price.py", "content": 'def price():\n\treturn "a\\\\b" + \'\\u00e9\' + "/" + "" + "&"\n'}, + ] + TWIN_SERIALIZED: ClassVar[str] = ( + '[{"name":"bundle.mthds","content":"domain = \\"x\\"\\n[concept.\u00c9t\u00e9]\\n' + 'description = \\"caf\u00e9 \u2014 \u00fcn\u00efc\u00f6d\u00e9 \u65e5\u672c \U0001f642\\"\\n"},' + '{"name":"funcs/price.py","content":"def price():\\n\\treturn \\"a\\\\\\\\b\\" + \'\\\\u00e9\' + \\"/\\" + \\"\\" + \\"&\\"\\n"}]' + ) + + # (topic, single-character content, whether the twin treats it as blank) + BLANK_PREDICATE: ClassVar[list[tuple[str, str, bool]]] = [ + ("U+0009 TAB", "\t", True), + ("U+000A LF", "\n", True), + ("U+000B VT", "\v", True), + ("U+000C FF", "\f", True), + ("U+000D CR", "\r", True), + ("U+0020 SPACE", " ", True), + ("U+00A0 NBSP", "\u00a0", True), + ("U+1680 OGHAM SPACE MARK", "\u1680", True), + ("U+2000 EN QUAD", "\u2000", True), + ("U+200A HAIR SPACE", "\u200a", True), + ("U+2028 LINE SEPARATOR", "\u2028", True), + ("U+2029 PARAGRAPH SEPARATOR", "\u2029", True), + ("U+202F NARROW NBSP", "\u202f", True), + ("U+205F MEDIUM MATHEMATICAL SPACE", "\u205f", True), + ("U+3000 IDEOGRAPHIC SPACE", "\u3000", True), + ("U+FEFF BOM — blank to the twin, not to str.isspace", "\ufeff", True), + ("U+001C FILE SEPARATOR — content to the twin, whitespace to str.isspace", "\x1c", False), + ("U+001D GROUP SEPARATOR — content to the twin, whitespace to str.isspace", "\x1d", False), + ("U+001E RECORD SEPARATOR — content to the twin, whitespace to str.isspace", "\x1e", False), + ("U+001F UNIT SEPARATOR — content to the twin, whitespace to str.isspace", "\x1f", False), + ("U+0085 NEXT LINE — content to the twin, whitespace to str.isspace", "\x85", False), + ("U+200B ZERO WIDTH SPACE — content on both sides", "\u200b", False), + ("U+180E MONGOLIAN VOWEL SEPARATOR — content on both sides", "\u180e", False), + ] + + # (topic, source) — every one read by the twin as "no files" + NO_FILES_SOURCES: ClassVar[list[tuple[str, str | None]]] = [ + ("the empty string", ""), + ("whitespace only", " "), + ("a mix of every whitespace kind, the BOM included", " \n\t\r \ufeff "), + ("None, a field never set", None), + ("the empty JSON array", "[]"), + ("the empty JSON array with trailing whitespace", "[] "), + ("the empty JSON array with leading whitespace", " []"), + ] + + # (topic, source) — every one refused by the twin with PipelineRequestError + CONTRACT_VIOLATIONS: ClassVar[list[tuple[str, str]]] = [ + ("raw bundle text is the legacy shape, not the catalog array", "domain = 'x'"), + ("a BOM before the array is not JSON on either side", "\ufeff[]"), + ("a JSON object, even one shaped like an entry", '{"name":"a.py","content":"x"}'), + ("JSON null", "null"), + ("a JSON string", '"abc"'), + ("a JSON number", "1"), + ("a bare NaN, refused by the decoder now and by the array check regardless", "NaN"), + ("an entry missing content", '[{"name":"a.py"}]'), + ("an entry that is a bare string", '["a.py"]'), + ("an entry that is null", "[null]"), + ("an entry that is an array", "[[]]"), + ("an entry whose name is not a string", '[{"name":1,"content":"x"}]'), + ("an entry whose content is not a string", '[{"name":"a","content":1}]'), + ("an entry whose content is null", '[{"name":"a","content":null}]'), + ("one good entry does not excuse a bad one", '[{"name":"a","content":"x"},{"name":"b"}]'), + # `json.loads` accepts these three as a Python extension and the twin's `JSON.parse` + # throws on every one. A bare `NaN` above is caught by the array check whatever the + # decoder does; inside an ignored member nothing downstream would ever look, so these + # are the cases that actually pin `parse_constant`. + ("NaN inside an ignored extra member", '[{"name":"a","content":"x","extra":NaN}]'), + ("Infinity inside an ignored extra member", '[{"name":"a","content":"x","extra":Infinity}]'), + ("-Infinity nested in an ignored member's array", '[{"name":"a","content":"x","extra":[1,-Infinity]}]'), + ("Infinity as a whole entry", "[Infinity]"), + ] + + # (topic, content, the exact string the twin's `JSON.stringify` printed for that one file) + # `json.dumps(ensure_ascii=False)` writes an unpaired surrogate raw, which is not UTF-8 + # encodable at all, so these fail at the storage boundary rather than at the typed surface. + LONE_SURROGATES: ClassVar[list[tuple[str, str, str]]] = [ + ("a lone high surrogate", "\ud800", '[{"name":"f","content":"\\ud800"}]'), + ("a lone low surrogate", "\udfff", '[{"name":"f","content":"\\udfff"}]'), + ("a lone surrogate between text", "a\ud800b", '[{"name":"f","content":"a\\ud800b"}]'), + ("an astral character stays raw, being one code point", "\U0001f642", '[{"name":"f","content":"\U0001f642"}]'), + ] + + # (topic, content, the exact string the twin's `JSON.stringify` printed for that one file) + # A Python `str` addresses code points and can hold the two halves of a pair as two of them, + # where the twin — addressing UTF-16 code units — holds one astral character and writes it + # raw. The pair is combined before the dump, so these are the twin's bytes and not two + # escapes. Built with `chr` so no source encoding can quietly collapse a pair into one code + # point, which is exactly the mistake that hides this case. + SURROGATE_PAIRS: ClassVar[list[tuple[str, str, str]]] = [ + ("a pair is the astral character it encodes", chr(0xD83D) + chr(0xDE42), '[{"name":"f","content":"\U0001f642"}]'), + ("the lowest pair", chr(0xD800) + chr(0xDC00), '[{"name":"f","content":"\U00010000"}]'), + ("the highest pair", chr(0xDBFF) + chr(0xDFFF), '[{"name":"f","content":"\U0010ffff"}]'), + ( + "a lone high before a pair stays lone, the pair still combines", + chr(0xD800) + chr(0xD800) + chr(0xDC00), + '[{"name":"f","content":"\\ud800\U00010000"}]', + ), + ("a low before a high is two lone surrogates", chr(0xDC00) + chr(0xD800), '[{"name":"f","content":"\\udc00\\ud800"}]'), + ( + "a pair followed by a lone low", + chr(0xD83D) + chr(0xDE42) + chr(0xDFFF), + '[{"name":"f","content":"\U0001f642\\udfff"}]', + ), + ] + + # A JSON integer long enough to exceed `sys.get_int_max_str_digits()` makes Python's decoder + # raise a bare `ValueError` from `int()` — not a `JSONDecodeError` — where the twin, having one + # number type and no such limit, parses the same bytes. The limit is 4300 digits by default. + OVERSIZED_INTEGER_DIGITS: ClassVar[int] = 4301 diff --git a/tests/unit/test_protocol_method_files.py b/tests/unit/test_protocol_method_files.py new file mode 100644 index 0000000..4fb971b --- /dev/null +++ b/tests/unit/test_protocol_method_files.py @@ -0,0 +1,193 @@ +"""The catalog serialization, held to the TypeScript twin edge for edge. + +`mthds/protocol/method_files.py` mirrors `mthds-js`'s `protocol/method_files.ts`, and a port of a +format like this one fails quietly: it round-trips its own output and still drops a file the twin +keeps, keeps one the twin drops, or writes bytes the twin would not. So the cases here are not what +the port was expected to do but what the twin was observed to do — `MethodFileCases` records the +bytes and the verdicts a run of the twin printed — and the happy round-trip is the least of them. +""" + +import json +import unicodedata + +import pytest +from pydantic import ValidationError + +from mthds.protocol.exceptions import PipelineRequestError +from mthds.protocol.method_files import _ECMASCRIPT_TRIM_CHARS, MethodFile, parse_method_files, serialize_method_files +from tests.unit.test_data import MethodFileCases + +_TWIN_FILES = [MethodFile.model_validate(entry) for entry in MethodFileCases.TWIN_FILES] + + +class TestMethodFiles: + def test_serialization_writes_the_twins_bytes(self): + # Same value is not the bar; same string is. `json.dumps` defaults — spaced separators, + # `\uXXXX` for anything outside ASCII — decode to the same files and store differently. + assert serialize_method_files(_TWIN_FILES) == MethodFileCases.TWIN_SERIALIZED + + def test_the_serialized_form_is_the_named_array_in_stored_order(self): + assert json.loads(serialize_method_files(_TWIN_FILES)) == MethodFileCases.TWIN_FILES + reversed_files = list(reversed(_TWIN_FILES)) + assert json.loads(serialize_method_files(reversed_files)) == list(reversed(MethodFileCases.TWIN_FILES)) + + def test_no_files_serializes_to_the_empty_string_and_never_to_the_literal_array(self): + # `""` is the platform's "no source" / "clear the field" sentinel. `"[]"` would be stored + # as a source; the platform does guard that literal explicitly and return no contents, so + # the reason for preferring `""` is that it is the sentinel the platform writes, not that + # `"[]"` would reach the TOML parser. + assert serialize_method_files([]) == "" + assert serialize_method_files([MethodFile(name="empty.py", content=" ")]) == "" + + def test_an_array_of_nothing_but_blank_files_collapses_to_the_sentinel(self): + # The round trip that reaches the sentinel, which is the whole reason the module exists: + # a catalog whose every entry is blank parses to no files and re-serializes to `""`. + source = json.dumps([{"name": "a.py", "content": ""}, {"name": "b.py", "content": "  "}]) + assert parse_method_files(source) == [] + assert serialize_method_files(parse_method_files(source)) == "" + + def test_serialization_drops_blank_entries_and_keeps_the_rest(self): + files = [MethodFile(name="a.py", content="x"), MethodFile(name="b.py", content=""), MethodFile(name="c.py", content="y")] + assert json.loads(serialize_method_files(files)) == [{"name": "a.py", "content": "x"}, {"name": "c.py", "content": "y"}] + + @pytest.mark.parametrize(("topic", "content", "is_blank"), MethodFileCases.BLANK_PREDICATE) + def test_blank_is_what_the_twin_calls_blank(self, topic: str, content: str, is_blank: bool): + # The predicate is ECMAScript `trim`'s set, spelled out. `str.strip()` would pass the + # happy cases and fail exactly these: the BOM it keeps, U+001C to U+001F and U+0085 it drops. + # Both directions go through a catalog array, because a bare control character is simply + # not valid JSON — asserting it is refused would prove nothing about the blank predicate. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + catalog = json.dumps([{"name": "f", "content": content}]) + if is_blank: + assert serialized == "", topic + assert parse_method_files(content) == [], topic + assert parse_method_files(catalog) == [], topic + else: + assert json.loads(serialized) == [{"name": "f", "content": content}], topic + assert parse_method_files(catalog) == [MethodFile(name="f", content=content)], topic + + def test_the_trim_set_is_ecmascripts_whole_trim_set(self): + # `BLANK_PREDICATE` samples code points; this derives the set, so a code point dropped + # from the middle of `_ECMASCRIPT_TRIM_CHARS` cannot pass the suite while silently + # reclassifying a file from blank to content. The authority is ECMAScript's own + # grammar — WhiteSpace (TAB, VT, FF, ZWNBSP, and every Space_Separator) plus the four + # LineTerminators — not either implementation's behaviour on a sample. + space_separators = {chr(code_point) for code_point in range(0x110000) if unicodedata.category(chr(code_point)) == "Zs"} + whitespace = space_separators | {chr(0x09), chr(0x0B), chr(0x0C), chr(0xFEFF)} + line_terminators = {chr(0x0A), chr(0x0D), chr(0x2028), chr(0x2029)} + assert set(_ECMASCRIPT_TRIM_CHARS) == whitespace | line_terminators + + @pytest.mark.parametrize(("topic", "content", "expected"), MethodFileCases.LONE_SURROGATES) + def test_an_unpaired_surrogate_is_escaped_as_the_twin_escapes_it(self, topic: str, content: str, expected: str): + # `ensure_ascii=False` writes an unpaired surrogate raw, and the resulting string cannot + # be encoded as UTF-8 at all — so it breaks at the storage boundary, not here, and the + # at-rest bytes differ from the twin's. The twin has escaped it since ES2019. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + assert serialized == expected, topic + assert serialized.encode("utf-8"), topic + assert parse_method_files(serialized) == [MethodFile(name="f", content=content)], topic + + def test_parsing_reads_the_twins_bytes_back_into_files(self): + assert parse_method_files(MethodFileCases.TWIN_SERIALIZED) == _TWIN_FILES + + @pytest.mark.parametrize(("topic", "source"), MethodFileCases.NO_FILES_SOURCES) + def test_a_blank_source_and_an_empty_array_are_both_no_files(self, topic: str, source: str | None): + assert parse_method_files(source) == [], topic + + def test_parsing_drops_blank_entries_so_the_round_trip_is_stable(self): + source = json.dumps([{"name": "a.py", "content": "x"}, {"name": "b.py", "content": " "}]) + assert parse_method_files(source) == [MethodFile(name="a.py", content="x")] + + def test_parsing_tolerates_and_strips_an_extra_member(self): + # The wire may carry more than the two members; the typed surface carries exactly two. + # Validating an entry through the closed model would refuse what the twin accepts. + assert parse_method_files('[{"name":"a.py","content":"x","extra":1}]') == [MethodFile(name="a.py", content="x")] + + def test_parsing_keeps_order_duplicates_and_blank_names(self): + # A blank name is not a reason to drop an entry — the platform's decoder disagrees, and the + # port follows the twin so the three implementations do not split two against one. + source = '[{"name":"z","content":"1"},{"name":"a","content":"2"},{"name":"a","content":"3"},{"name":"","content":"4"}]' + assert parse_method_files(source) == [ + MethodFile(name="z", content="1"), + MethodFile(name="a", content="2"), + MethodFile(name="a", content="3"), + MethodFile(name="", content="4"), + ] + + def test_round_trip_serialize_then_parse(self): + assert parse_method_files(serialize_method_files(_TWIN_FILES)) == _TWIN_FILES + assert parse_method_files(serialize_method_files([])) == [] + + @pytest.mark.parametrize(("topic", "source"), MethodFileCases.CONTRACT_VIOLATIONS) + def test_anything_but_the_named_array_is_a_contract_violation(self, topic: str, source: str): + with pytest.raises(PipelineRequestError) as exc_info: + parse_method_files(source) + assert "method file" in str(exc_info.value).lower(), topic + + def test_a_source_that_exhausts_the_decoder_is_a_contract_violation(self): + # `json.loads` raises `RecursionError` rather than `JSONDecodeError` on a source nested + # deeper than its decoder allows, and `RecursionError` is not a `ValueError`, so catching + # only `JSONDecodeError` let it escape this typed surface. + # + # The depth that trips it is an interpreter build constant, not `sys.getrecursionlimit()`: + # it rose by an order of magnitude in CPython 3.14 and differs even between patch + # releases, so pinning one depth pins an accident of one build. Depth 10000 refuses on + # 3.11 to 3.13 and parses cleanly on 3.14, which is a supported interpreter the CI matrix + # runs — so the depth is discovered here instead. + # + # This guard is best-effort and NOT a parity rule: the twin's `JSON.parse` is iterative + # and accepts every depth tried, so it is the module's one forced divergence from it. + for depth in (10_000, 200_000, 500_000): + source = '[{"name":"a","content":"x","extra":' + "[" * depth + "]" * depth + "}]" + try: + json.loads(source) + except RecursionError: + with pytest.raises(PipelineRequestError): + parse_method_files(source) + return + pytest.skip("no tried nesting depth exhausts this interpreter's JSON decoder") + + @pytest.mark.parametrize(("topic", "content", "expected"), MethodFileCases.SURROGATE_PAIRS) + def test_a_surrogate_pair_is_written_as_the_astral_character_the_twin_writes(self, topic: str, content: str, expected: str): + # Python addresses code points and so can spell an astral character two ways; the twin + # addresses UTF-16 code units and has only one spelling. Escaping both halves instead of + # combining them makes the at-rest bytes differ from the twin's on every one of the pairs, + # and costs serialization its idempotence: the parse combines what the dump split. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + assert serialized == expected, topic + assert serialized.encode("utf-8"), topic + assert serialize_method_files(parse_method_files(serialized)) == expected, topic + + def test_the_surrogate_rules_cover_the_name_as_well_as_the_content(self): + # One `json.dumps` writes both members, so both get the same treatment — pinned here + # because a fix applied to only one of them would still pass every content-only case. + pair = chr(0xD83D) + chr(0xDE42) + assert serialize_method_files([MethodFile(name=pair, content="x")]) == '[{"name":"\U0001f642","content":"x"}]' + lone = "a" + chr(0xD800) + "b" + assert serialize_method_files([MethodFile(name=lone, content="x")]) == '[{"name":"a\\ud800b","content":"x"}]' + + def test_an_oversized_integer_in_an_ignored_member_is_accepted_as_the_twin_accepts_it(self): + # Python runs every JSON integer through `int()`, which refuses a literal longer than + # `sys.get_int_max_str_digits()` with a bare `ValueError` — neither a `JSONDecodeError` nor + # a `RecursionError` — so it escaped this typed surface entirely, while the twin parsed the + # same bytes and stripped the member. Integers are decoded as floats here, which is the + # twin's own number model and immune to the limit. + source = '[{"name":"a.py","content":"x","extra":' + "1" * MethodFileCases.OVERSIZED_INTEGER_DIGITS + "}]" + # The vector is only meaningful while Python's own decoder still refuses these digits. + with pytest.raises(ValueError, match="Exceeds the limit"): + json.loads(source) + assert parse_method_files(source) == [MethodFile(name="a.py", content="x")] + + def test_an_oversized_integer_where_a_string_belongs_is_still_a_contract_violation(self): + # Accepting the number must not accept the entry: the twin refuses this one too. + source = '[{"name":"a.py","content":' + "1" * MethodFileCases.OVERSIZED_INTEGER_DIGITS + "}]" + with pytest.raises(PipelineRequestError): + parse_method_files(source) + + def test_the_typed_shape_is_closed_and_strict(self): + # Leniency lives in the parser, not the type: a consumer constructing a file gets exactly + # `name` and `content`, both strings. + with pytest.raises(ValidationError): + MethodFile.model_validate({"name": "a.py", "content": "x", "extra": 1}) + with pytest.raises(ValidationError): + MethodFile.model_validate({"name": 1, "content": "x"}) From a2c670417f7bc4b3d1661df2f9a03843540b2ab2 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Sat, 12 Sep 2026 22:34:58 +0200 Subject: [PATCH 09/11] =?UTF-8?q?feature/Run-source-predicates=20=C2=B7=20?= =?UTF-8?q?L-260907-f4e16c=20=E2=80=94=20the=20rule=20belongs=20to=20the?= =?UTF-8?q?=20request=20(#68)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The run-source rule — which combinations of `pipe_code`, `mthds_contents`, the pipelex-api bundle encodings (`files` / `bundle_b64`), a published address (`method_ref`) and the hosted catalog id (`method_id`) may be sent together — now has one Python definition in `mthds.protocol.options`, the twin of `mthds-js`'s `protocol/options.ts`, instead of living as private helpers inside `pipelex-sdk`'s client where `mthds`' own client saw none of it. Those definitions and their tests are the whole of it: nothing in this package enforces from them, so `MthdsAPIClient` keeps forwarding the keys inside `extra` without interpreting them and no client behaviour changes. The first consumer is `PipelexAPIClient` in `pipelex-sdk`, which pins an older `mthds` and can replace its private copies once a release ships the module. Whether the generic any-runner client should enforce Pipelex extension rules at all, and whether the standard's package is where the hosted selector rules belong, is the open design question L-260912-1b10c1 — the branch no longer depends on its answer. Closes L-260907-f4e16c 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 1 + docs/runners.md | 21 ++- mthds/protocol/options.py | 242 +++++++++++++++++++++++++ mthds/runners/api/client.py | 46 ++++- tests/unit/test_api_runner_protocol.py | 54 ++++++ tests/unit/test_protocol_options.py | 212 ++++++++++++++++++++++ 6 files changed, 568 insertions(+), 8 deletions(-) create mode 100644 mthds/protocol/options.py create mode 100644 tests/unit/test_protocol_options.py diff --git a/CHANGELOG.md b/CHANGELOG.md index ea5a123..2bc268a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ ### Added - **`mthds.protocol.method_files` — the catalog serialization of a stored method's source**: `MethodFile` (`name` + `content`) and the pair `serialize_method_files` / `parse_method_files` between a list of files and the JSON `[{ name, content }]` string the hosted platform persists for a method's `.mthds` source and its custom PipeFunc `python`. The empty list serializes to `""`, the platform's "no source" sentinel, never to `"[]"`; blank-content entries are dropped in both directions; anything but the named array raises `PipelineRequestError`, including the three constants Python's `json` accepts and `JSON.parse` refuses. It mirrors the `mthds` npm package's `protocol/method_files.ts` edge for edge — blank is ECMAScript-blank, and the bytes are `JSON.stringify`'s down to both surrogate rules, an unpaired one escaped and an adjacent pair combined into the astral character Python alone can spell two ways — so a Python client reads a stored method's bundle from here rather than re-porting the platform's decoder. Python's own number model is kept out of the way too: integers decode as floats, the twin's single number type, so a literal longer than `sys.get_int_max_str_digits()` cannot raise a bare `ValueError` through the typed surface. One divergence from the twin is forced rather than chosen and is documented as such: Python's decoder recurses where `JSON.parse` iterates, so a deeply nested source is refused here and accepted there, at a depth that is an interpreter build constant. +- **`mthds.protocol.options` — the run-source argument surface:** predicates that decide which combinations of `pipe_code`, `mthds_contents`, the bundle encodings `files` / `bundle_b64`, `method_ref` and `method_id` may be sent together (`assert_exclusive_run_sources`, `has_bundle_payload`, `assert_method_ref_pairs_with_nothing`), plus the selector normalization `normalized_selector` / `run_selector_extensions`, which refuses a non-string selector. Nothing in this package calls them, so no client behaviour changes: `MthdsAPIClient` still forwards the keys inside `extra` without interpreting them. ## [v0.14.0] - 2026-09-06 diff --git a/docs/runners.md b/docs/runners.md index d5ed60d..cc81158 100644 --- a/docs/runners.md +++ b/docs/runners.md @@ -6,7 +6,7 @@ The `mthds` package is the Python client for the open-source `pipelex-api` runne The protocol contract and its implementations live in separate packages: -- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `method_files.py` (the catalog serialization of a stored method's source — see [The catalog serialization](#the-catalog-serialization-method_files) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). +- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `method_files.py` (the catalog serialization of a stored method's source — see [The catalog serialization](#the-catalog-serialization-method_files) below), `options.py` (the run-source argument surface — see [Run sources: what may be combined](#run-sources-what-may-be-combined) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). - `mthds/runners/` — every runner implementation, one subpackage per runner: - `api/` — the API runner: `client.py` (`MthdsAPIClient`, one file with its helpers), `models.py` (the Dict-serialized wire models — `DictConcept`, `DictStuffAbstract`, `DictWorkingMemoryAbstract`, `DictPipeOutputAbstract`, `DictRunResultExecute` — the runners' concrete JSON materialization of the protocol's domain shapes), `exceptions.py` (API auth + the protocol's 202-degrade error, `RunStillRunningError`). - `pipelex/runner.py` — `PipelexRunner`, the local runner that shells out to the `pipelex` CLI. @@ -142,6 +142,25 @@ The abstract `MTHDSProtocol` interface carries the protocol's **basic** argument - Extension args never appear in this SDK — not even as convenience params. They ride the generic `extra` mapping on both `execute` and `start`: `client.start(pipe_code="answer", extra={"some_server_arg": True})` merges `some_server_arg` into the request body as a top-level property. The server you call defines and handles its own extension args; consult that server's API documentation for what it accepts. - Protocol args inside `extra` are rejected client-side with `PipelineRequestError` — pass them as named parameters. +### Run sources: what may be combined + +A run request has to name something to run, and several of the ways to name it exclude each other. That rule is an invariant of the request itself rather than of any one runner, so it lives in `mthds.protocol.options` for any Python client to enforce from one place instead of re-deriving it. **The definitions are there; no client enforces from them yet** — see "What is actually wired" below. The module is the twin of `mthds-js/src/protocol/options.ts`, whose two exported predicates it matches message for message; it is **not** a twin of either server, so a rejection here can read differently from the 422 it pre-empts (`pipelex-api` and the hosted platform each word these refusals their own way, and the platform checks the two exclusivity arms in the opposite order — `pipelex-api` has no two-encodings check at all). + +Three layers of argument meet in it: the protocol's own source (`pipe_code`, `mthds_contents`), the pipelex-api extensions that carry a whole method bundle (`files`, `bundle_b64`) or its published address (`method_ref`), and the hosted platform's catalog id (`method_id`). Only the first layer is a named parameter of `execute` / `start`; the others ride `extra` here, and are named parameters on a client that types its own stack's arguments. + +| Function | What it decides | +| --- | --- | +| `assert_exclusive_run_sources` | A bundle is self-contained, so it cannot be combined with `mthds_contents`, and `files` / `bundle_b64` are two encodings of one bundle. Keys off **presence** (`files={}` beside a zip is still two encodings), while `mthds_contents` counts only when non-empty. | +| `has_bundle_payload` | Whether a **runnable** bundle is present — an empty map or string carries no method, so it satisfies nothing. | +| `assert_method_ref_pairs_with_nothing` | A `method_ref` is a complete run source: exclusive with `mthds_contents`, with a bundle encoding, and with `method_id`. `pipe_code` beside it is legal (it overrides the fetched manifest's `main_pipe`), and inline source + `method_id` stays legal (the inline source runs, the id becomes run-history linkage). | +| `normalized_selector` / `run_selector_extensions` | The boundary normalization a selector goes through: a non-string is refused rather than silently dropped or forwarded to a server 422, and an absent or empty selector contributes nothing. | + +#### What is actually wired + +The table above describes the predicates, not a guarantee any client makes: **nothing in this package calls them.** `MthdsAPIClient.execute` / `start` check only that something to run was named — a pipe, inline contents, or any extension arg — and forward the keys inside `extra` without interpreting them. That is the client's documented contract: it serves any MTHDS-compliant runner, and which extension arg names a run source is that runner's to define. So an illegal combination sent through `extra` still reaches the server and comes back as its own 422, exactly as it did before this module existed. + +The predicates are there for a client that takes these arguments by name, which is where the combination is knowable before the request leaves. `PipelexAPIClient` in `pipelex-sdk` is the first such client; it still carries its own private copies and pins an older `mthds`, so replacing them waits on a release that ships this module. Which layer should enforce the extension rules — the generic client here, the client that names the arguments, or a Pipelex-scoped module beside this one — is still an open question, and the `pipe-selector` campaign will change this same surface after it is settled. + ## The durable run lifecycle (hosted API only) — lives in `pipelex-sdk` `start` (a protocol route) returns a `pipeline_run_id` only; turning that id into a result means polling the run to completion. That **durable run lifecycle is not part of the MTHDS Protocol** — it is a hosted-API extension, so it no longer lives in this package. It is exposed by `PipelexAPIClient` in `pipelex-sdk` (`get_run_status` / `get_run_result` / `wait_for_result` / `start_and_wait`), which builds on this protocol base. A bare runner serves no run store; the SDK detects that via the `version()` handshake and falls back to a blocking `execute`. diff --git a/mthds/protocol/options.py b/mthds/protocol/options.py new file mode 100644 index 0000000..008df7b --- /dev/null +++ b/mthds/protocol/options.py @@ -0,0 +1,242 @@ +"""The MTHDS Protocol's run-source argument surface — which combinations of the `execute` / +`start` arguments name something to run, and which are illegal. + +The protocol has no request *model*: a runner takes the request's basic arguments as named +parameters and serializes the wire body directly, merging any server-specific extension args +(`extra`) into the body as top-level properties. The Python expression of that argument surface +is therefore not a class but this set of predicates over the arguments themselves — each taking +plain values, so a caller passes them from wherever its own signature keeps them (a named +parameter, or a key it pulled out of `extra`). + +They travel with the request rather than with any one runner: which source combinations are +legal is an invariant of the request itself, so a Python client that builds one can enforce it +from this single definition instead of re-deriving it and drifting. **This module is that +definition, and nothing in the Python stack enforces from it yet.** `MthdsAPIClient` in this +package calls none of these predicates: it is a client for any MTHDS-compliant runner, so it +forwards the keys inside `extra` without interpreting them and leaves the addressed server to +answer for the combination. `PipelexAPIClient` in `pipelex-sdk` is the client that takes these +arguments by name and can pre-empt that refusal, and it still carries its own private copies — +it pins an older `mthds` and cannot import this module until one ships with it. + +Which layer should enforce the extension rules — the generic client, the client that names the +arguments, or a Pipelex-scoped module beside this one — is still open. These definitions are the +part that is settled. + +This module is the Python twin of `mthds-js/src/protocol/options.ts`, whose two exported +predicates it matches message for message. It is **not** a twin of either server, and a +rejection here can therefore read differently from the 422 it pre-empts: `pipelex-api` phrases +the bundle-versus-contents refusal its own way, and the platform validator tests the two +exclusivity arms in the opposite order, so one illegal body can draw a different sentence from +each. The JS twin is the binding parity; the servers are the inspiration. + +Three layers of argument meet here, and each predicate's docstring says which layer it is about: + +- **the protocol's own run source** — `pipe_code`, `mthds_contents`; +- **the pipelex-api extensions** that carry a whole method bundle (`files`, `bundle_b64`) or its + published address (`method_ref`); +- **the hosted platform's** catalog id (`method_id`). + +Only the first layer is a named parameter of `MTHDSProtocol.execute` / `start`; the other two +reach the wire through the generic `extra` passthrough, or as named parameters of a client that +types its own stack's arguments. All three are stated here on one argument: a rule split across +the packages that enforce it is a rule that drifts, which is the situation today and the one +this module exists to end. + +The `pipe-selector` campaign will change this surface (a `pipe_ref` beside `pipe_code`, and +which of the two may be combined with what). When it does, it changes it here. +""" + +from collections.abc import Mapping +from typing import Final + +from mthds.protocol.exceptions import PipelineRequestError + +RUN_ARG_FILES: Final[str] = "files" +"""Wire key of the pipelex-api `files` extension — a whole method bundle as a path-to-text map.""" + +RUN_ARG_BUNDLE_B64: Final[str] = "bundle_b64" +"""Wire key of the pipelex-api `bundle_b64` extension — the same bundle as a base64 zip.""" + +RUN_ARG_METHOD_REF: Final[str] = "method_ref" +"""Wire key of the pipelex-api `method_ref` extension — a published method's address.""" + +RUN_ARG_METHOD_ID: Final[str] = "method_id" +"""Wire key of the hosted platform's `method_id` run arg — a stored method's catalog id.""" + + +def normalized_selector(*, name: str, value: object) -> str | None: + """Normalize one method-selector argument at a client's boundary. + + A **non-string** value is refused rather than dropped or forwarded. A published client + validates its request-option types at its own boundary, so that one wrong value gets one + answer, rather than a bare truthiness check silently dropping the falsy wrong types (`0`, + `[]`) and forwarding the truthy ones (`123`, `["mt_1"]`) to a server `422`. This is a + deliberate DIVERGENCE from the JS lineage rather than parity with it: `nonEmptyString` in + `pipelex-sdk-js/src/client.ts` never throws, so it drops most wrong types silently and + forwards a `["mt_1"]` whose `length` happens to be non-zero. Refusing at the boundary is the + better behaviour, but the two sides do not agree on it yet. `value` is typed `object` rather + than `str | None` deliberately — this helper *is* the runtime boundary, and the callers it + guards against are the untyped ones a type checker never sees. + + An absent or **empty** value normalizes to `None`: an empty selector selects nothing, so it + is not sent and does not satisfy a client's "something to run" precondition. + + Args: + name: The argument's name, as it appears in the error message and on the wire. + value: The caller-supplied value, of any type. + + Returns: + The selector string, or None when it is absent or empty. + + Raises: + PipelineRequestError: If the value is present and is not a string. + """ + if value is None: + return None + if not isinstance(value, str): + msg = f"{name} must be a string, received {type(value).__name__}." + raise PipelineRequestError(msg) + return value or None + + +def run_selector_extensions(*, method_ref: object = None, method_id: object = None) -> dict[str, str]: + """Normalize the method selectors a client names into the extension mapping they ride to the + wire. + + The layering seam: `method_ref` (layer 2 — the Pipelex API's run source, resolved by the + runner) and `method_id` (layer 3 — the hosted platform's run arg) are named parameters on + the client that types its own stack's arguments, and each reaches the wire as a top-level + body property through the protocol client's generic extension mechanism, which merges it + without knowing what it means. + + Both go through `normalized_selector`, so a non-string is refused and an absent or empty + value contributes nothing. Rejecting the caller's own reserved argument names inside `extra` + is the calling client's business — it owns that list — and merging is left to the caller too, + so this returns only what the selectors contribute. + + Args: + method_ref: The published method's address, or None. + method_id: The hosted catalog id, or None. + + Returns: + The selector contribution to the extension mapping — empty when neither is present. + + Raises: + PipelineRequestError: If a selector is present and is not a string. + """ + extensions: dict[str, str] = {} + selected_method_ref = normalized_selector(name=RUN_ARG_METHOD_REF, value=method_ref) + if selected_method_ref is not None: + extensions[RUN_ARG_METHOD_REF] = selected_method_ref + selected_method_id = normalized_selector(name=RUN_ARG_METHOD_ID, value=method_id) + if selected_method_id is not None: + extensions[RUN_ARG_METHOD_ID] = selected_method_id + return extensions + + +def has_bundle_payload(*, files: Mapping[str, str] | None = None, bundle_b64: str | None = None) -> bool: + """Does the request carry a method bundle (the pipelex-api `files` / `bundle_b64` extension)? + + A bundle satisfies a client's "something to run" precondition on its own — it carries its + own `.mthds`, so neither `pipe_code` nor `mthds_contents` is required alongside it. Unlike + `assert_exclusive_run_sources`, this keys off a RUNNABLE payload: an empty map or string + carries no method, so it satisfies nothing (and must not be sent — the runner rejects a + zero-file bundle). + + Args: + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + + Returns: + True when either encoding carries a method. + """ + return bool(files) or bool(bundle_b64) + + +def assert_exclusive_run_sources( + *, + mthds_contents: list[str] | None = None, + files: Mapping[str, str] | None = None, + bundle_b64: str | None = None, +) -> None: + """Enforce the run-source exclusivity contract shared by every client: a method bundle is + self-contained (`files` / `bundle_b64` carry their own `.mthds`), so it cannot be combined + with `mthds_contents`, and `files` / `bundle_b64` are two encodings of one bundle. + + Exclusivity keys off PRESENCE, not emptiness — a caller who supplies `files={}` alongside + `bundle_b64` still expressed two encodings — while `mthds_contents` counts only when + non-empty (an empty list is "no contents"). Both servers count these values the same way, + but the wording is the `mthds-js` twin's, byte for byte, and neither server's. + + Args: + mthds_contents: Inline MTHDS bundle contents, or None. + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + + Raises: + PipelineRequestError: If two run sources that exclude each other are both present. + """ + has_files = files is not None + has_zip = bundle_b64 is not None + has_contents = bool(mthds_contents) + if has_files and has_zip: + msg = "files and bundle_b64 are two encodings of the same bundle and are mutually exclusive; provide one." + raise PipelineRequestError(msg) + if (has_files or has_zip) and has_contents: + msg = "A method bundle (files/bundle_b64) is self-contained; it cannot be combined with mthds_contents." + raise PipelineRequestError(msg) + + +def assert_method_ref_pairs_with_nothing( + *, + method_ref: object = None, + mthds_contents: list[str] | None = None, + files: Mapping[str, str] | None = None, + bundle_b64: str | None = None, + method_id: object = None, +) -> None: + """Enforce the run routes' `method_ref` exclusivity, so an illegal pairing fails before + anything hits the wire. + + A `method_ref` is a complete run source (the fetched package carries its `.mthds` and its + entry pipe), so it pairs with NOTHING: not with inline `mthds_contents`, not with a bundle + encoding, and not with the hosted `method_id` — an address run has its own provenance and + needs no linkage id. + + The one documented run-route exception is deliberately NOT here: inline source + `method_id` + stays legal (the inline source runs; the id demotes to run-history linkage). `pipe_code` + beside a `method_ref` is legal too — it overrides the fetched manifest's `main_pipe`. + `mthds_contents` counts when non-empty and a bundle encoding counts when the key is present, + as on both servers. A selector counts only when non-empty, which is not the servers' rule: + they declare `min_length=1`, so an empty selector is a `422` there rather than absent. The + messages are those of the private `assertMethodRefPairsWithNothing` in + `pipelex-sdk-js/src/client.ts`, byte for byte; `mthds-js` exports no counterpart. + + Args: + method_ref: The published method's address, or None — when absent or empty, no pairing + is checked, though both selectors are still refused if present and not a string. + mthds_contents: Inline MTHDS bundle contents, or None. + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + method_id: The hosted catalog id, or None. + + Raises: + PipelineRequestError: If a `method_ref` is paired with any other run source, or if a + selector is present and is not a string. + """ + selected_method_ref = normalized_selector(name=RUN_ARG_METHOD_REF, value=method_ref) + selected_method_id = normalized_selector(name=RUN_ARG_METHOD_ID, value=method_id) + if selected_method_ref is None: + return + if mthds_contents: + msg = "method_ref and inline mthds_contents are mutually exclusive; send one or the other." + raise PipelineRequestError(msg) + if files is not None or bundle_b64 is not None: + msg = "method_ref and a method bundle (bundle_b64 / files) are mutually exclusive; send one or the other." + raise PipelineRequestError(msg) + if selected_method_id is not None: + msg = ( + "method_ref and method_id are mutually exclusive: an address run carries its own provenance " + "and takes no run-history linkage id. Send exactly one method selector." + ) + raise PipelineRequestError(msg) diff --git a/mthds/runners/api/client.py b/mthds/runners/api/client.py index 66e16cb..df3e97f 100644 --- a/mthds/runners/api/client.py +++ b/mthds/runners/api/client.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any, ClassVar, Self, cast +from typing import TYPE_CHECKING, Any, ClassVar, Literal, Self, cast from urllib.parse import quote import httpx @@ -161,12 +161,12 @@ async def execute( Complete execution results including run state and output Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a + protocol arg. RunStillRunningError: If the server answers 202 (the protocol's optional async degrade) — the run continues server-side; resume by `run_id`. """ - if not pipe_code and not mthds_contents and not extra: - msg = "Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API execute." - raise PipelineRequestError(msg) + _assert_run_sources(pipe_code=pipe_code, mthds_contents=mthds_contents, extra=extra, route="execute") body = _build_run_body( pipe_code=pipe_code, @@ -214,10 +214,12 @@ async def start( RunResultStart — the authoritative server-generated `pipeline_run_id` (no output yet). On a hosted deployment the id is durable — poll the durable run lifecycle (a hosted extension, exposed by `pipelex-sdk`). + + Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a + protocol arg. """ - if not pipe_code and not mthds_contents and not extra: - msg = "Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API start." - raise PipelineRequestError(msg) + _assert_run_sources(pipe_code=pipe_code, mthds_contents=mthds_contents, extra=extra, route="start") body = _build_run_body( pipe_code=pipe_code, @@ -388,6 +390,36 @@ def _raise_if_execute_degraded(self, response: httpx.Response) -> None: _VALIDATION_RESULT_ADAPTER: TypeAdapter[ValidationResult] = TypeAdapter(ValidationResult) +def _assert_run_sources( + *, pipe_code: str | None, mthds_contents: list[str] | None, extra: dict[str, Any] | None, route: Literal["execute", "start"] +) -> None: + """Guard the run-source arguments of one `/execute` | `/start` call, before any body is built. + + One check: something to run must have been named — a pipe, inline contents, or a + server-specific extension arg. Which extension arg names a run source, and which of them + exclude each other, is the server's own contract: this client serves any MTHDS-compliant + runner, so it counts the keys inside `extra` without interpreting them, and the runner + answers for the combination it was sent. A client that takes those arguments by name can + pre-empt the refusal from the shared definitions in `mthds.protocol.options`. + + The extension mapping is built here rather than only in `_build_run_body` so that a protocol + arg smuggled through `extra` is refused at this boundary, before any body exists. + + Args: + pipe_code: The pipe selector, or None. + mthds_contents: Inline MTHDS bundle contents, or None. + extra: Server-specific extension args from the caller, or None. + route: The protocol route being called, for the error message (`execute` / `start`). + + Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a protocol arg. + """ + extensions = _build_extensions(extra) + if not pipe_code and not mthds_contents and not extensions: + msg = f"Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API {route}." + raise PipelineRequestError(msg) + + def _build_run_body( *, pipe_code: str | None, diff --git a/tests/unit/test_api_runner_protocol.py b/tests/unit/test_api_runner_protocol.py index af5c591..d41d5f5 100644 --- a/tests/unit/test_api_runner_protocol.py +++ b/tests/unit/test_api_runner_protocol.py @@ -2,6 +2,7 @@ import asyncio import inspect +from typing import Any import httpx import pytest @@ -11,6 +12,7 @@ from mthds.protocol.models import InvalidValidationReport, ModelCategory, ModelDeck, ValidationReport, VersionInfo from mthds.protocol.protocol import MTHDSProtocol from mthds.runners.api.client import MthdsAPIClient +from tests.unit.test_data import ExecuteWireResponses _BASE_URL = "http://localhost:8081" @@ -198,3 +200,55 @@ def test_version_parses_handshake(self, mocker: MockerFixture) -> None: assert info.runner_version == "0.3.0" # Implementation identification passes through as extensions, never named by the SDK. assert info.model_extra == {"some_vendor_name": "vendor-runner"} + + # ── execute / start run sources ─────────────────────────────── + + @pytest.mark.parametrize( + ("route", "wire_response"), + [("execute", ExecuteWireResponses.REDUCED), ("start", {"pipeline_run_id": "run_7f3a"})], + ) + def test_run_requires_something_to_run(self, route: str, wire_response: dict[str, Any], mocker: MockerFixture) -> None: + """Neither a pipe, nor contents, nor an extension arg: the call is refused before any request. + + The transport stub answers with a well-formed body on purpose. Were the guard removed, + the call would run to completion and this test would report the guard's absence — rather + than a parse error from a stub that could never have produced a result anyway. + """ + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=wire_response))) + with pytest.raises(PipelineRequestError, match=f"must be provided to the API {route}"): + asyncio.run(getattr(client, route)()) + send_mock.assert_not_called() + + @pytest.mark.parametrize( + ("route", "wire_response"), + [("execute", ExecuteWireResponses.REDUCED), ("start", {"pipeline_run_id": "run_7f3a"})], + ) + def test_run_forwards_extension_args_without_interpreting_them(self, route: str, wire_response: dict[str, Any], mocker: MockerFixture) -> None: + """`extra` is the server's to define, so this client forwards its keys without judging them. + + The combination sent here — inline contents beside both bundle encodings at once — is + illegal on `pipelex-api`, which is the runner that defines `files` and `bundle_b64`. They + are that runner's extension args and not the protocol's, and `MthdsAPIClient` is + documented as a client for any MTHDS-compliant runner, so it is the addressed server that + answers for the combination. Pre-empting the refusal is the business of a client that + takes those arguments by name, from the shared definitions in `mthds.protocol.options`. + """ + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=wire_response))) + + asyncio.run(getattr(client, route)(mthds_contents=['domain = "answer"'], extra={"files": {}, "bundle_b64": "UEsDBA=="})) + + sent = send_mock.call_args.kwargs["content"].decode("utf-8") + assert '"mthds_contents":["domain = \\"answer\\""]' in sent + assert '"files":{}' in sent + assert '"bundle_b64":"UEsDBA=="' in sent + + def test_execute_accepts_a_bundle_as_the_only_run_source(self, mocker: MockerFixture) -> None: + """A bundle in `extra` satisfies the precondition and merges into the body as a top-level property.""" + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=ExecuteWireResponses.REDUCED))) + + asyncio.run(client.execute(extra={"files": {"main.mthds": 'domain = "answer"'}})) + sent = send_mock.call_args.kwargs["content"].decode("utf-8") + assert '"files":{"main.mthds":"domain = \\"answer\\""}' in sent diff --git a/tests/unit/test_protocol_options.py b/tests/unit/test_protocol_options.py new file mode 100644 index 0000000..5b2083a --- /dev/null +++ b/tests/unit/test_protocol_options.py @@ -0,0 +1,212 @@ +"""Unit tests for the run-source argument surface — the predicates every Python client shares.""" + +import pytest + +from mthds.protocol.exceptions import PipelineRequestError +from mthds.protocol.options import ( + RUN_ARG_BUNDLE_B64, + RUN_ARG_FILES, + RUN_ARG_METHOD_ID, + RUN_ARG_METHOD_REF, + assert_exclusive_run_sources, + assert_method_ref_pairs_with_nothing, + has_bundle_payload, + normalized_selector, + run_selector_extensions, +) + +_ADDRESS = "github.com/Pipelex/methods/documents@v0.1.0" + + +class TestRunSourceOptions: + """The run-source predicates and the selector normalization that goes with them.""" + + # ── wire keys ──────────────────────────────────────────────── + + def test_keys_match_the_wire(self) -> None: + """The constants are the request's own property names — the runners read `extra` by them.""" + assert (RUN_ARG_FILES, RUN_ARG_BUNDLE_B64, RUN_ARG_METHOD_REF, RUN_ARG_METHOD_ID) == ( + "files", + "bundle_b64", + "method_ref", + "method_id", + ) + + # ── normalized_selector ────────────────────────────────────── + + def test_absent_selector_is_none(self) -> None: + """An absent selector normalizes to None.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value=None) is None + + def test_empty_selector_is_none(self) -> None: + """An empty selector selects nothing, so it normalizes to None rather than riding the wire.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value="") is None + + def test_present_selector_passes_through(self) -> None: + """A non-empty string is returned unchanged.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value=_ADDRESS) == _ADDRESS + + @pytest.mark.parametrize("wrong_value", [123, ["mt_1"], 0, [], {}, True]) + def test_non_string_selector_is_refused(self, wrong_value: object) -> None: + """A non-string is refused rather than dropped (falsy) or forwarded (truthy) — one answer + for every wrong type. + + This DIVERGES from the JS lineage rather than matching it: `nonEmptyString` in + `pipelex-sdk-js` never throws, so it drops most of these silently and forwards + `["mt_1"]`, whose `length` happens to be non-zero. Refusing is the better behaviour; + the two sides do not agree on it yet. + """ + with pytest.raises(PipelineRequestError, match="must be a string, received"): + normalized_selector(name=RUN_ARG_METHOD_REF, value=wrong_value) + + def test_selector_error_names_the_argument_and_the_type(self) -> None: + """The message names the argument the caller passed and the type they passed for it.""" + with pytest.raises(PipelineRequestError, match=r"method_id must be a string, received int\."): + normalized_selector(name=RUN_ARG_METHOD_ID, value=7) + + # ── run_selector_extensions ────────────────────────────────── + + def test_no_selector_contributes_nothing(self) -> None: + """Neither selector present: the contribution is empty, so the caller's `extra` is untouched.""" + assert run_selector_extensions() == {} + + def test_both_selectors_ride_under_their_wire_keys(self) -> None: + """Each selector reaches the mapping under its own wire key.""" + assert run_selector_extensions(method_ref=_ADDRESS, method_id="mt_abc") == { + RUN_ARG_METHOD_REF: _ADDRESS, + RUN_ARG_METHOD_ID: "mt_abc", + } + + def test_empty_selectors_are_dropped(self) -> None: + """An empty selector contributes nothing (presence is decided after normalization).""" + assert run_selector_extensions(method_ref="", method_id="") == {} + + def test_extensions_refuse_a_non_string_selector(self) -> None: + """The normalization is the same one `normalized_selector` applies.""" + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + run_selector_extensions(method_id=42) + + # ── has_bundle_payload ─────────────────────────────────────── + + def test_no_bundle(self) -> None: + """Neither encoding present: nothing to run from a bundle.""" + assert has_bundle_payload() is False + + def test_files_payload(self) -> None: + """A non-empty file map carries a method.""" + assert has_bundle_payload(files={"main.mthds": 'domain = "answer"'}) is True + + def test_zip_payload(self) -> None: + """A non-empty base64 zip carries a method.""" + assert has_bundle_payload(bundle_b64="UEsDBA==") is True + + def test_empty_encodings_carry_no_method(self) -> None: + """Unlike exclusivity, this keys off a RUNNABLE payload: empty encodings carry nothing.""" + assert has_bundle_payload(files={}, bundle_b64="") is False + + # ── assert_exclusive_run_sources ───────────────────────────── + + def test_nothing_present_is_legal(self) -> None: + """Exclusivity says nothing about an absent source — the precondition is a separate check.""" + assert_exclusive_run_sources() + + def test_contents_alone_is_legal(self) -> None: + """Inline contents alone is the protocol's own run source.""" + assert_exclusive_run_sources(mthds_contents=['domain = "answer"']) + + def test_files_alone_is_legal(self) -> None: + """One bundle encoding alone is legal.""" + assert_exclusive_run_sources(files={"main.mthds": 'domain = "answer"'}) + + def test_two_encodings_are_refused(self) -> None: + """`files` and `bundle_b64` are two encodings of one bundle.""" + with pytest.raises(PipelineRequestError, match="two encodings of the same bundle"): + assert_exclusive_run_sources(files={"main.mthds": "x"}, bundle_b64="UEsDBA==") + + def test_two_encodings_key_off_presence_not_emptiness(self) -> None: + """An empty `files` map beside a zip still expressed two encodings.""" + with pytest.raises(PipelineRequestError, match="two encodings of the same bundle"): + assert_exclusive_run_sources(files={}, bundle_b64="UEsDBA==") + + def test_bundle_with_contents_is_refused(self) -> None: + """A bundle carries its own `.mthds`, so it cannot be combined with inline contents.""" + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], files={"main.mthds": "x"}) + + def test_zip_with_contents_is_refused(self) -> None: + """The same rule holds for the zip encoding.""" + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], bundle_b64="UEsDBA==") + + def test_empty_contents_beside_a_bundle_is_legal(self) -> None: + """An empty contents list is "no contents", so it does not collide with a bundle.""" + assert_exclusive_run_sources(mthds_contents=[], files={"main.mthds": "x"}) + + def test_bundle_with_contents_keys_off_presence_not_emptiness(self) -> None: + """An empty `files` map beside inline contents is still two run sources. + + This arm and `has_bundle_payload` deliberately read the same value differently: an empty + encoding is PRESENT for exclusivity and RUNNABLE for neither. Pinning both sides of that + disagreement is what stops a later refactor from routing exclusivity through + `has_bundle_payload` and turning this refusal into an accepted request. + """ + assert has_bundle_payload(files={}) is False + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], files={}) + + # ── assert_method_ref_pairs_with_nothing ───────────────────── + + def test_absent_method_ref_checks_nothing(self) -> None: + """With no address, every other combination is somebody else's rule.""" + assert_method_ref_pairs_with_nothing(mthds_contents=['domain = "answer"'], method_id="mt_abc") + + def test_empty_method_ref_checks_nothing(self) -> None: + """An empty address selects nothing, so it is not an address run.""" + assert_method_ref_pairs_with_nothing(method_ref="", mthds_contents=['domain = "answer"'], method_id="mt_abc") + + def test_method_ref_alone_is_legal(self) -> None: + """The address alone is a complete run source.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS) + + def test_method_ref_with_contents_is_refused(self) -> None: + """An address and inline contents are two run sources.""" + with pytest.raises(PipelineRequestError, match="inline mthds_contents are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, mthds_contents=['domain = "answer"']) + + def test_method_ref_with_files_is_refused(self) -> None: + """An address and a bundle are two run sources — the arm the private SDK copy lacked.""" + with pytest.raises(PipelineRequestError, match=r"a method bundle \(bundle_b64 / files\) are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, files={"main.mthds": "x"}) + + def test_method_ref_with_zip_is_refused(self) -> None: + """The bundle arm keys off presence, so the zip encoding is refused too.""" + with pytest.raises(PipelineRequestError, match=r"a method bundle \(bundle_b64 / files\) are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, bundle_b64="UEsDBA==") + + def test_method_ref_with_method_id_is_refused(self) -> None: + """An address run has its own provenance and takes no run-history linkage id.""" + with pytest.raises(PipelineRequestError, match="takes no run-history linkage id"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, method_id="mt_abc") + + def test_method_ref_with_empty_method_id_is_legal(self) -> None: + """An empty id selects nothing, so it is not a second selector.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, method_id="") + + def test_empty_contents_beside_method_ref_is_legal(self) -> None: + """An empty contents list is "no contents" — the presence semantics the server uses.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, mthds_contents=[]) + + def test_non_string_method_ref_is_refused(self) -> None: + """The address goes through the same boundary normalization as any selector.""" + with pytest.raises(PipelineRequestError, match="method_ref must be a string"): + assert_method_ref_pairs_with_nothing(method_ref=["github.com/Pipelex/methods"]) + + def test_non_string_method_id_is_refused_without_an_address(self) -> None: + """The selector type refusal does not depend on an unrelated argument. + + `method_id` is normalized before the no-address early return, so one wrong value gets one + answer whether or not a `method_ref` accompanies it — the unconditional guarantee + `docs/runners.md` advertises for a selector at this boundary. + """ + with pytest.raises(PipelineRequestError, match=r"method_id must be a string, received int\."): + assert_method_ref_pairs_with_nothing(method_id=42) From 30455391c221a00efadcc928c9d813254623c493 Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 18 Sep 2026 17:54:46 +0200 Subject: [PATCH 10/11] =?UTF-8?q?feature/Trim-concept-abstract=20=C2=B7=20?= =?UTF-8?q?L-260918-8fba7a=20=E2=80=94=20trim=20ConceptAbstract,=20seriali?= =?UTF-8?q?ze=20the=20ref=20(#69)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ConceptAbstract` keeps `code`, `domain_code` and the `concept_ref` property; `description`, `structure_class_name` and `refines` leave it, because none of them is a fact about a stuff — a concept's definition belongs to the library the method loads, and a runtime that needs one of them in memory declares it on its own subclass. `StuffAbstract` gains a field serializer on `concept`, so every dump of a stuff emits the standard's wire form, `{"concept": ".", "content": …}`, instead of leaving each runner to reduce the object itself. This is the rule the MTHDS spec landed in the CLI I/O contract's "Stuffs on the Wire" section. It is a deliberate break: a stuff, a working memory or a pipe output can no longer be validated back from its own dump, and a consumer that did so resolves the concept by ref through the loaded library instead — the changelog, the docs and the serializer's docstring all say so. Closes L-260918-8fba7a 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018GodVX8dbGwhAZGWnuz6M3 --- ## Summary by cubic Trims `ConceptAbstract` to the reference surface — `code`, `domain_code` and `concept_ref` — and makes `StuffAbstract` serialize `concept` as the ref string, so every dump of a stuff emits the standard wire form `{"concept": ".", "content": …}` instead of leaving each runner to reduce the object itself. The definition fields (`description`, `structure_class_name`, `refines`) no longer travel with a concept; they belong to the library the method loads, and a runtime that needs them declares them on its own subclass. The two override points reach different code: `concept_ref` on the concept reaches every reduction including the serializer, while `serialize_concept` on the stuff reaches a dump alone — `DictRunResultExecute.from_pipe_output` reduces by `concept_ref`, so a `serialize_concept` override is not followed there (which point it should honour is filed as L-260918-5f09d8). Closes L-260918-8fba7a. **Breaking changes** - A stuff, working memory, or pipe output can no longer be validated back from its own dump — a consumer that re-validated a dump resolves the concept by ref through the loaded library instead. - Producing a dependency-contributed concept's crate key `::.` is a subclass's job: override `StuffAbstract.serialize_concept` under that name. Written for commit 7732e1b7ca571c5a1b853eddaf536a064ed60087. Summary will update on new commits. Review in cubic --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 4 ++ docs/runners.md | 2 + mthds/protocol/concept.py | 20 ++++++-- mthds/protocol/stuff.py | 30 +++++++++++- mthds/runners/api/models.py | 3 ++ tests/unit/test_api_runner_models.py | 43 ++++++++++------ tests/unit/test_protocol_stuff.py | 73 ++++++++++++++++++++++++++++ 7 files changed, 156 insertions(+), 19 deletions(-) create mode 100644 tests/unit/test_protocol_stuff.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bc268a..bef094d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ - **`mthds.protocol.method_files` — the catalog serialization of a stored method's source**: `MethodFile` (`name` + `content`) and the pair `serialize_method_files` / `parse_method_files` between a list of files and the JSON `[{ name, content }]` string the hosted platform persists for a method's `.mthds` source and its custom PipeFunc `python`. The empty list serializes to `""`, the platform's "no source" sentinel, never to `"[]"`; blank-content entries are dropped in both directions; anything but the named array raises `PipelineRequestError`, including the three constants Python's `json` accepts and `JSON.parse` refuses. It mirrors the `mthds` npm package's `protocol/method_files.ts` edge for edge — blank is ECMAScript-blank, and the bytes are `JSON.stringify`'s down to both surrogate rules, an unpaired one escaped and an adjacent pair combined into the astral character Python alone can spell two ways — so a Python client reads a stored method's bundle from here rather than re-porting the platform's decoder. Python's own number model is kept out of the way too: integers decode as floats, the twin's single number type, so a literal longer than `sys.get_int_max_str_digits()` cannot raise a bare `ValueError` through the typed surface. One divergence from the twin is forced rather than chosen and is documented as such: Python's decoder recurses where `JSON.parse` iterates, so a deeply nested source is refused here and accepted there, at a depth that is an interpreter build constant. - **`mthds.protocol.options` — the run-source argument surface:** predicates that decide which combinations of `pipe_code`, `mthds_contents`, the bundle encodings `files` / `bundle_b64`, `method_ref` and `method_id` may be sent together (`assert_exclusive_run_sources`, `has_bundle_payload`, `assert_method_ref_pairs_with_nothing`), plus the selector normalization `normalized_selector` / `run_selector_extensions`, which refuses a non-string selector. Nothing in this package calls them, so no client behaviour changes: `MthdsAPIClient` still forwards the keys inside `extra` without interpreting them. +### Changed + +- **`ConceptAbstract` names a concept and nothing more, and `StuffAbstract` serializes `concept` as the ref string (Breaking)**: the protocol's concept model keeps `code`, `domain_code` and the `concept_ref` property — `description`, `structure_class_name` and `refines` are gone, since a concept's definition belongs to the library the method loads and never travels beside a stuff, and a runtime that needs one of them in memory declares it on its own subclass. `StuffAbstract` now carries a field serializer on `concept`, so every dump of a stuff emits `{"concept": ".", "content": …}` — the wire form the standard's [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/) requires — instead of leaving each runner to reduce the object itself. The dependency-contributed crate key `::.` is a subclass's to produce, by overriding `StuffAbstract.serialize_concept` under that name: the protocol model holds no declaring package address. The dump is one-way, which is the point rather than an oversight — a `StuffAbstract`, a `WorkingMemoryAbstract` or a `PipeOutputAbstract` can no longer be validated back from its own dump, since the definition such a parse would need is exactly what no longer travels. A consumer that used to re-validate a dump resolves the concept by ref through the library the method loads, the way the input side already does. Which of the two override points a runtime picks decides what this SDK's own output path emits: `DictRunResultExecute.from_pipe_output` reduces a stuff by `concept_ref`, so a `concept_ref` override reaches it and a `serialize_concept` override does not. + ## [v0.14.0] - 2026-09-06 ### Added diff --git a/docs/runners.md b/docs/runners.md index cc81158..da6c53e 100644 --- a/docs/runners.md +++ b/docs/runners.md @@ -64,6 +64,8 @@ async with MthdsAPIClient() as client: The Dict wire models are extension-open at every level, matching the protocol's OpenAPI spec (which declares `pipe_output` as an open object). Base fields are typed; anything more a runner returns rides `model_extra`. The hosted `pipelex-api` runner dumps its full `PipeOutput` — per-stuff `stuff_code` / `stuff_name`, pipe-output `graph_spec` / `tokens_usages` / `working_memory_raw` and assembly errors — and all of it is preserved. A stuff's `concept` accepts both wire forms: the reduced namespaced ref string (what this SDK's own serialization emits) or the full concept object (`DictConcept`, what the hosted runner dumps); `DictStuffAbstract.concept_ref` normalizes either form to the ref string. +On the producing side the protocol's own domain shapes say the same thing, and say it once: `ConceptAbstract` carries the reference alone (`code` + `domain_code`, with `concept_ref` joining them), and `StuffAbstract` serializes its `concept` field through that property, so every dump of a stuff emits `{"concept": ".", "content": …}` without a runner having to remember to reduce it. That is the standard's rule — a concept's definition (its description, its structure, what it refines, any runtime class name) belongs to the library the method loads and never travels beside a stuff, per the "Stuffs on the Wire" section of the [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/). A runtime that needs more of the definition in memory declares it on its own `ConceptAbstract` subclass, where it stays out of the wire form. Two consequences are worth naming. The crate key for a concept a dependency contributes is `::.`, and the protocol model holds no declaring package address, so producing that form is a subclass's business — it overrides `StuffAbstract.serialize_concept` under that name, since a second `field_serializer` for the field under any other name is a class-definition error and a serializer re-annotated onto the field is ignored silently. Which override point a runtime picks decides what this SDK's own output path emits, since `DictRunResultExecute.from_pipe_output` reduces a stuff by `concept_ref` rather than through the serializer: a `concept_ref` override reaches it, a `serialize_concept` override does not. And the dump is one-way: these shapes are strict, so a stuff, a working memory or a pipe output cannot be validated back from what it dumped, the definition such a parse would need being precisely what no longer travels. A consumer that re-validated a dump resolves the concept by ref through the library the method loads instead, as the input side already does; the `Dict` wire models above are the other way to read one back, since they accept either form. + ### The validate artifacts: `pipe_io_contracts` and `input_form` Two artifacts of the valid `/validate` report are owned by the standard since `mthds` v0.9.0 — [Pipe I/O Contracts](https://mthds.ai/latest/spec/pipe-io-contracts/) and the [Input-Form Descriptor](https://mthds.ai/latest/spec/input-form-descriptor/) — and this package types them: `mthds/protocol/pipe_io_contracts.py` and `mthds/protocol/input_form.py` mirror those pages exactly, snake_case slot for snake_case slot. Both are **recommended extension fields** of the report, not base fields: the protocol's base did not change, and how a caller asks a runner for the descriptor is implementation-defined (the hosted Pipelex API gates it behind its `views` request extension). They therefore arrive beside the report's typed base fields, and you narrow them by declaring them as typed fields on a model that extends the report — pydantic parses the maps and the discriminated union from the plain annotations, with no adapter machinery (this is exactly how `pipelex-sdk`'s report narrowing consumes them): diff --git a/mthds/protocol/concept.py b/mthds/protocol/concept.py index 692f08a..d1c71fd 100644 --- a/mthds/protocol/concept.py +++ b/mthds/protocol/concept.py @@ -7,14 +7,28 @@ class ConceptAbstract(BaseModel, ABC): + """A concept as a stuff names it — a reference, and nothing more. + + The standard puts a concept's definition — its description, its structure, + what it refines, and anything an implementation attaches to it such as the + name of a runtime class — in the library the method loads, never beside a + stuff (see the "Stuffs on the Wire" section of the + [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/)). A runtime + that needs more of the definition in memory declares it on its own subclass. + """ + model_config = ConfigDict(extra="forbid", strict=True) code: str domain_code: str - description: str - structure_class_name: str - refines: str | None = None @property def concept_ref(self) -> str: + """The domain-qualified reference, `.`. + + This is the crate key for a concept the method's own package declares and + for a native one. A concept a dependency contributes is keyed + `::.`, and this model holds no declaring + package address, so that form is a subclass's to produce. + """ return f"{self.domain_code}.{self.code}" diff --git a/mthds/protocol/stuff.py b/mthds/protocol/stuff.py index 350fd8c..6adfdb9 100644 --- a/mthds/protocol/stuff.py +++ b/mthds/protocol/stuff.py @@ -1,7 +1,7 @@ from abc import ABC from typing import Any, Generic, TypeVar -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, field_serializer from mthds.protocol.concept import ConceptType @@ -17,6 +17,34 @@ class StuffAbstract(BaseModel, ABC, Generic[ConceptType, StuffContentType]): concept: ConceptType content: StuffContentType + @field_serializer("concept") + def serialize_concept(self, concept: ConceptType) -> str: + """Emit the concept as its reference string. + + A stuff names its concept and never carries its definition, so every + dump of a stuff — whatever a runtime keeps on its own concept subclass — + is the standard's wire form. + + The dump is therefore one-way: these models are strict, so a stuff, a + working memory or a pipe output cannot be validated back from its own + dump. That is the point rather than an oversight — the definition the + parse would need is not on the wire, and a reader resolves the ref + through the concept library the method loads, the way the input side + already does. + + A subclass that emits something else — the `::.` + crate key of a dependency-contributed concept, say — has two ways in, and + they do not reach the same places. Overriding `concept_ref` on the concept + reaches every reduction, this serializer included. Overriding this method + reaches a dump and nothing else: it must keep **this name**, decorated + again or as a plain method, since a second `field_serializer` for + `concept` under another name raises pydantic's `multiple-field-serializers` + error at class definition, while re-annotating the field with a serializer + or putting a `model_serializer` on the concept subclass is ignored in + favour of this one with no diagnostic at all. + """ + return concept.concept_ref + class StuffContentAbstract(BaseModel, ABC): pass diff --git a/mthds/runners/api/models.py b/mthds/runners/api/models.py index d12832f..17000cb 100644 --- a/mthds/runners/api/models.py +++ b/mthds/runners/api/models.py @@ -113,6 +113,9 @@ def _serialize_working_memory(cls, working_memory: WorkingMemoryAbstract[StuffTy # Convert each Stuff -> DictStuff by dumping only the content for stuff_name, stuff in working_memory.root.items(): + # The concept is reduced by `concept_ref`, which is what the protocol's own field + # serializer delegates to. A runtime that overrides `serialize_concept` instead is + # not followed here: which override point this path should honour is L-260918-5f09d8. dict_stuff = cls._dict_stuff_class( concept=stuff.concept.concept_ref, content=stuff.content.model_dump(serialize_as_any=True), diff --git a/tests/unit/test_api_runner_models.py b/tests/unit/test_api_runner_models.py index f56b6d5..b2b989e 100644 --- a/tests/unit/test_api_runner_models.py +++ b/tests/unit/test_api_runner_models.py @@ -2,39 +2,44 @@ from typing import Any -from pydantic import BaseModel - +from mthds.protocol.concept import ConceptAbstract +from mthds.protocol.pipe_output import PipeOutputAbstract +from mthds.protocol.stuff import StuffAbstract, StuffContentAbstract +from mthds.protocol.working_memory import WorkingMemoryAbstract from mthds.runners.api.models import MAIN_STUFF_NAME, DictRunResultExecute -class _StubContent(BaseModel): +class _StubContent(StuffContentAbstract): value: str -class _StubConcept(BaseModel): - concept_ref: str +class _StubConcept(ConceptAbstract): + pass -class _StubStuff(BaseModel): - concept: _StubConcept - content: _StubContent +class _StubStuff(StuffAbstract[_StubConcept, _StubContent]): + pass -class _StubWorkingMemory(BaseModel): - root: dict[str, _StubStuff] - aliases: dict[str, str] +class _StubWorkingMemory(WorkingMemoryAbstract[_StubStuff]): + pass -class _StubPipeOutput(BaseModel): - working_memory: _StubWorkingMemory - pipeline_run_id: str +class _StubPipeOutput(PipeOutputAbstract[_StubWorkingMemory]): + pass def _pipe_output_with_run_id(run_id: str) -> Any: """A minimal pipe-output stub carrying a non-empty pipeline_run_id.""" return _StubPipeOutput( working_memory=_StubWorkingMemory( - root={"main": _StubStuff(concept=_StubConcept(concept_ref="answer.Answer"), content=_StubContent(value="42"))}, + root={ + "main": _StubStuff( + stuff_code="stuff_1", + concept=_StubConcept(code="Answer", domain_code="answer"), + content=_StubContent(value="42"), + ) + }, aliases={MAIN_STUFF_NAME: "main"}, ), pipeline_run_id=run_id, @@ -77,3 +82,11 @@ def test_main_stuff_name_extension_passes_through(self) -> None: assert result.model_extra is not None assert result.model_extra["main_stuff_name"] == "main" + + def test_concept_is_reduced_to_its_ref_string(self) -> None: + """The wire model carries the concept as `.` — the reduction this path + performs is `concept_ref`, the same property the protocol's field serializer delegates to. + """ + result = DictRunResultExecute.from_pipe_output(pipe_output=_pipe_output_with_run_id("run_1")) + + assert result.pipe_output.working_memory.root["main"].concept == "answer.Answer" diff --git a/tests/unit/test_protocol_stuff.py b/tests/unit/test_protocol_stuff.py new file mode 100644 index 0000000..c40bda2 --- /dev/null +++ b/tests/unit/test_protocol_stuff.py @@ -0,0 +1,73 @@ +"""Unit tests for the protocol's stuff and concept shapes — what a stuff says about its concept.""" + +import json + +import pytest +from pydantic import ValidationError + +from mthds.protocol.concept import ConceptAbstract +from mthds.protocol.stuff import StuffAbstract, StuffContentAbstract + + +class _StubConcept(ConceptAbstract): + """A runtime's concept: the protocol's two fields, plus whatever the runtime needs in memory.""" + + structure_class_name: str + + +class _StubContent(StuffContentAbstract): + value: str + + +class _StubStuff(StuffAbstract[_StubConcept, _StubContent]): + pass + + +def _stub_stuff() -> _StubStuff: + return _StubStuff( + stuff_code="stuff_1", + stuff_name="analysis", + concept=_StubConcept(code="ContractAnalysis", domain_code="legal", structure_class_name="ContractAnalysis"), + content=_StubContent(value="high"), + ) + + +class TestStuffConceptSerialization: + def test_concept_abstract_carries_the_reference_only(self) -> None: + """A concept's definition belongs to the library, so the protocol model holds the two fields + that name it and nothing else. + """ + assert set(ConceptAbstract.model_fields) == {"code", "domain_code"} + + def test_definition_fields_are_refused(self) -> None: + """The trimmed fields are not silently tolerated: the model is closed, so a payload + carrying a definition fails the parse rather than smuggling it beside a stuff. + """ + with pytest.raises(ValidationError, match="description"): + ConceptAbstract.model_validate({"code": "ContractAnalysis", "domain_code": "legal", "description": "A contract analysis"}) + + def test_concept_ref_is_the_domain_qualified_reference(self) -> None: + assert _stub_stuff().concept.concept_ref == "legal.ContractAnalysis" + + def test_dump_emits_the_concept_as_its_reference(self) -> None: + """`model_dump()` gives the ref string, not the concept object — including what the + runtime's own subclass carries beside the two protocol fields. + """ + dumped = _stub_stuff().model_dump() + + assert dumped["concept"] == "legal.ContractAnalysis" + assert dumped["content"] == {"value": "high"} + assert dumped["stuff_code"] == "stuff_1" + assert dumped["stuff_name"] == "analysis" + + def test_json_dump_emits_the_concept_as_its_reference(self) -> None: + """JSON mode agrees with python mode — the wire form is the protocol's on both.""" + assert json.loads(_stub_stuff().model_dump_json())["concept"] == "legal.ContractAnalysis" + + def test_serialization_schema_declares_the_concept_a_string(self) -> None: + """A server embedding a stuff in a response publishes `concept` as a string, so the + generated OpenAPI says what the wire actually carries. + """ + schema = _StubStuff.model_json_schema(mode="serialization") + + assert schema["properties"]["concept"] == {"type": "string"} From de5bb939f82e5462b5d12adf21e23cc976278efc Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Fri, 18 Sep 2026 22:20:11 +0200 Subject: [PATCH 11/11] Release v0.15.0 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018GodVX8dbGwhAZGWnuz6M3 --- CHANGELOG.md | 2 +- pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bef094d..df0f40d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -## [Unreleased] +## [v0.15.0] - 2026-09-18 ### Added diff --git a/pyproject.toml b/pyproject.toml index e30259f..bcfb4da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "mthds" -version = "0.14.0" +version = "0.15.0" description = "The Python interface for methods — base structures for structured outputs and the base runner for executing methods via API." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] diff --git a/uv.lock b/uv.lock index 32ba765..e9c7857 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.14.0" +version = "0.15.0" source = { editable = "." } dependencies = [ { name = "httpx" },