diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index d0497a0..e47ccff 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -1,183 +1,83 @@ --- name: release description: > - Automates the mthds-python release workflow: bumps the version in - pyproject.toml, finalizes the CHANGELOG.md Unreleased section, runs quality - checks, regenerates uv.lock, creates a release/vX.Y.Z branch, commits, pushes, - and opens a PR to main. Use when user says "release", "cut a release", "bump - version", "prepare a release", "make a release", "ship it", "create release - branch", or any variation of shipping a new version of the mthds Python - package. The user can optionally provide changelog content inline when invoking - the skill (e.g. "/release Added a new base runner option"), which will be used - as the changelog entry for this version. + Cut a release of mthds-python, the `mthds` Python package published to PyPI: + the release/vX.Y.Z worktree, the pyproject.toml bump and the uv.lock that + follows, the changelog entry, the quality gates, the corpus parity check + against mthds-js, one commit, and a pull request to main. Use when the user + says "release", "cut a release", "bump version", "prepare a release", "make a + release", "ship it", "create release branch", "promote dev to main", "tag a + version", or any variation of shipping a new version of the mthds Python + package. Changelog content passed inline ("/release Added a base runner + option") becomes the entry. The merge is landed by /ledger-land, never by this + skill. --- -# mthds-python Release Workflow +# Releasing mthds-python -This skill handles the full release cycle for the `mthds` Python package (the -`mthds-python` repo). A release is a `release/vX.Y.Z` branch that PRs into -`main`; merging to `main` triggers `publish.yml`, which builds the wheel, -publishes it to PyPI as `mthds`, and creates the GitHub release from the -changelog notes. +The procedure is the workspace release play, [`docs/releasing.md`](../../../../docs/releasing.md) at the workspace root — read it first, then run it with what follows. The repo key is `mthds-python`, the base is `dev`, and the pull request targets `main`: `guard-branches.yml`'s `gate-main` job refuses any head branch but `release/vX.Y.Z` into `main`, so there is no other way in. The release worktree is `_mthds-python--release`, made with `wt add mthds-python release --branch release/vX.Y.Z`. The repo declares neither `.worktree.toml` nor `.worktreeinclude`, so `wt` resolves the base from `origin/dev` and provisions with the Makefile's `install` target, which is what creates the `.venv` every gate below runs out of. -## Files touched +## What ships -- **`pyproject.toml`** — the `version` field (line 3) -- **`CHANGELOG.md`** — add `[vX.Y.Z] - YYYY-MM-DD` entry (remove `[Unreleased]` if present) -- **`uv.lock`** — regenerated via `make li` (lock + install) +The merge to `main` publishes, from `publish.yml`, which fires on the push (`on: push: branches: [main]`) — so the run is keyed to `main` and its `headSha` is the merge commit itself: -## Workflow +- **The `mthds` package on PyPI.** The `build` job runs `python3 -m build`, and the `publish-to-pypi` job uploads the dists by trusted publishing into the `pypi` environment, . The upload passes no `skip-existing`, so a push to `main` carrying a version PyPI already holds fails there — and nothing else stands between an unbumped push and that failure, since `version-check.yml` runs on pull requests only. +- **The GitHub Release and the `vX.Y.Z` tag**, by the same workflow's `github-release` job, which needs both of the others. The notes are the changelog section for the version read out of `pyproject.toml` — everything from its `## [vX.Y.Z] - ` heading down to **two** lines above the next `## [v…] - ` heading, with blank lines dropped and each line's leading and trailing whitespace stripped — handed to `gh release create` alongside `--generate-notes`. The slice stops two lines short rather than one because the workflow subtracts a line twice, so the line sitting directly above the next heading never reaches the notes: harmless while that line is the blank separator the changelog format puts there, and a silently missing last bullet if an entry is ever written flush against the heading below it. Keep the blank line between sections. When no such heading is found the step warns, leaves the notes empty and exits 0, so the Release ships carrying the bare line `Release vX.Y.Z` instead of failing. The tag exists only as a side effect of `gh release create`; nothing in the workflow runs `git tag -a`, so always pass `--tags` when reading the tags with `git describe`. +- **Nothing else.** No documentation site is deployed: `docs/` ships inside the repo and no workflow publishes it anywhere. -### 1. Pre-flight checks +Sigstore signing runs **before** the Release is created and carries no `continue-on-error`, so a signing outage leaves the version published on PyPI with no tag and no GitHub Release at all. The cure is to re-run that run's **failed jobs**, which keeps the successful upload; a fresh full run would fail at the upload over the files already on PyPI. No workflow in this repo declares `workflow_dispatch`, so the run the push to `main` created is the only run there is. -- Read the current version from `pyproject.toml`. -- Read `CHANGELOG.md` to understand the current state. -- Run `git status` and `git log origin/main..HEAD` to assess the working tree: - - If there are **uncommitted changes** (staged or unstaged), warn the user and - ask whether to commit them as part of the release, stash them, or abort. - - If there are **unpushed commits** on the current branch, list them so the - user is aware — these will be included in the release branch. +The landing verifies the publish — the run, the registry's answer, the tag: -### 2. Determine the bump type - -Ask the user which kind of version bump they want — **patch**, **minor**, or -**major** — unless they already specified it. Show the current version and what -the new version would be for each option so the choice is concrete. - -While the package is pre-1.0 (`0.y.z`), treat the `0.MINOR.PATCH` segments the -way the project has been using them: a breaking change bumps the minor, a -backward-compatible feature or fix bumps the patch. If the changelog for this -release contains a `### Breaking Changes` section, steer the user toward at least -a minor bump. - -### 3. Run quality checks - -Run `make agent-check`. This is the gate — if it fails, stop and report the -errors so they can be fixed before retrying. Do not proceed past this step on -failure. - -### 4. Ensure we're on the right branch - -The release branch must be named `release/vX.Y.Z` where X.Y.Z is the **new** -version — `guard-branches.yml` rejects any other source branch merging into -`main`, and `version-check.yml` rejects a mismatch between the branch name and -the `pyproject.toml` version. All file modifications (changelog, version bump, -lock) must happen on this branch. - -- If already on `release/vX.Y.Z` matching the new version, stay on it. -- If on `dev`, `main`, or any other branch, create and switch to - `release/vX.Y.Z` from the current HEAD. -- If on a `release/` branch for a **different** version, warn the user and ask - how to proceed. - -### 5. Finalize the changelog - -Add a new version entry at the top of the changelog for the release. - -1. If there is an `## [Unreleased]` section, **remove it** (including any blank - lines that follow it) and replace it with the new version heading. Any - content that was under `[Unreleased]` becomes the content of the new version. -2. If there is no `[Unreleased]` section, insert the new version heading - directly after the `# Changelog` title. -3. **Never add an `[Unreleased]` heading.** The changelog should only contain - concrete version entries. -4. If the user provided changelog content when invoking the skill (e.g. - `/release Added a new base runner option`), **merge** that content with any - existing `[Unreleased]` content (do not discard either source). Format the - combined content properly under the appropriate headings — this repo uses - `### Breaking Changes`, `### Added`, `### Changed`, `### Fixed`, `### Removed` - — inferring headings from the content when possible. -5. If the release has no changelog content yet (neither from an `[Unreleased]` - section nor from inline user input), ask the user what to include before - proceeding. -6. The result should look like: - -```markdown -# Changelog - -## [vX.Y.Z] - YYYY-MM-DD - -### Changed -- ... - -## [vPREVIOUS] - PREVIOUS-DATE -... +```bash +gh run list --workflow=publish.yml --branch main --limit 3 --json conclusion,headSha,url # the run whose headSha is the merge commit: success +curl -s https://pypi.org/pypi/mthds/json | jq -r .info.version # the registry's answer: X.Y.Z +git fetch --tags --prune origin && git tag --list vX.Y.Z # the tag ``` -### 6. Bump the version in pyproject.toml - -Edit `pyproject.toml` line 3 to the new version string. Only change the version -field — don't touch anything else. +`gh release view vX.Y.Z` confirms the Release and its notes. The registry can equally be read with `pip index versions mthds` where a `pip` is on the PATH — the worktree's uv-made venv has none. -### 7. Lock dependencies +## Version files and the lock -Run `make li` to regenerate `uv.lock` and reinstall. This ensures the lockfile -reflects the new version in `pyproject.toml`. The `package-check.yml` CI job runs -`uv lock --locked` and fails the PR if `uv.lock` is out of sync, so this step is -not optional. If it fails, stop and report the error. +- **`pyproject.toml`** — the `[project]` table's `version`, and nothing else in the file. Keep it the file's **first** `version = ` line and keep it starting at column zero: `changelog-check.yml` and `publish.yml` both read it with `grep -m 1 'version = '`, and `version-check.yml` with `grep '^version'`. +- **`uv.lock`** — regenerated by `make li` (`make lock` then `make install`, that is `uv lock` followed by `uv sync --all-extras`), run after the bump so the lockfile records the new number. If it fails, stop and report it rather than committing a stale lock: `package-check.yml` runs `uv lock --locked` and diffs the file on every pull request. +- **Also stamped: nothing.** The release number lives in `pyproject.toml` alone — no `__version__`, no README badge, no literal in `docs/`. The other version numbers the package declares are **not** this one, and a release never touches them: `MTHDS_STANDARD_VERSION` in `mthds/package/manifest/schema.py` and `PROTOCOL_VERSION` in `mthds/protocol/protocol.py` are copies of cuts made by the MTHDS standard, moved when the standard moves and never because this library is shipping. `docs/versioning.md` is the account of every number in play and which of them this repo owns. -### 8. Commit and push +## Gates -Stage all release-related changes. This includes at minimum `pyproject.toml`, -`CHANGELOG.md`, and `uv.lock`, plus any other files the user chose to include in -step 1 (e.g. previously uncommitted work that belongs in this release). +Run in the worktree, in this order, before the commit: -Commit with the message: +1. **`make agent-check`** — unused imports, ruff format, ruff lint, pyright, mypy. **It rewrites files** (`fix-unused-imports`, `format`, and `lint`, which is `ruff check --fix`), so whatever it touches joins the release commit. These are the fixing counterparts of exactly what `lint-check.yml` runs read-only on the pull request — `merge-check-ruff-format`, `merge-check-ruff-lint`, `merge-check-pyright`, `merge-check-mypy` — across every supported Python, so a red one here is a red pull request there. Red blocks the release: fix the code, never loosen the target. `make check` additionally runs pylint, which no workflow does; it is not part of this flow. +2. **`make agent-test`** — the pytest suite, quiet unless it fails. `tests-check.yml` runs `make gha-tests` (`pytest --exitfirst --quiet`) on every supported Python, so a failure here is the pull request's failure too. +3. **The protocol fixture corpus, against `mthds-js`.** `tests/fixtures/protocol/` is committed byte-identically in this repo and in `mthds-js`, and `protocol-corpus-parity.yml` checks the sibling out on every pull request and fails on any difference, so a release promoting a one-sided recapture opens red. Read the sibling's checkout before committing, with the workflow's own exclusions: -``` -Release vX.Y.Z -``` - -Push the branch to origin with `-u` to set up tracking. + ```bash + diff --recursive --brief -x README.md -x '*.fixture.ts' tests/fixtures/protocol ../mthds-js/tests/fixtures/protocol + ``` -### 9. Open a PR + That reads `mthds-js`'s main checkout, which under the main-checkout invariant sits on `dev` — the branch CI falls back to — so bring it current before trusting a green result, and treat the CI job as the authority. A difference is never repaired on the release branch: the capture is regenerated from `pipelex` and copied into both mirrors at once, which is the `projection-corpus-update` skill's job and a separate pull request in each repo, and the release waits for both. The mechanics are `tests/fixtures/protocol/README.md`, and the cross-repo authority is `conformance/scripts/check-protocol-fixture-parity.py`. -Create a pull request targeting `main` with: +## The release commit -- **Title:** `Release vX.Y.Z` -- **Body:** Include: - - The changelog entries for this version (copied from CHANGELOG.md) - - A note about the version bump from old to new +`pyproject.toml`, `uv.lock`, `CHANGELOG.md`, and each file `make agent-check` rewrote — staged by name. -Use this format for the PR body: +## CI on the release pull request -```markdown -## Release vX.Y.Z +- **`guard-branches.yml`** (`gate-main`) — the head branch into `main` matches `^release/v[0-9]+\.[0-9]+\.[0-9]+$` exactly. Its `protect-workflows` job additionally refuses a change under `.github/workflows/` from an author whose effective repository permission is not `write` or `admin`. +- **`version-check.yml`** — the `pyproject.toml` version equals the version in the branch name. A head that does not match the release form does not slip past it: the first step calls a non-match "skipping" and exits 0, but that ends the step alone, and the comparison in the last step then runs against an empty branch version and fails. +- **`changelog-check.yml`** — `CHANGELOG.md` carries a `## [vX.Y.Z] - ` heading for the version in `pyproject.toml`. It asserts nothing about `[Unreleased]`; leaving none behind is the play's rule, not CI's. +- **`package-check.yml`** (`uv-lock-check`, on every pull request) — `uv lock --locked` leaves `uv.lock` unchanged. +- **`lint-check.yml`** — the merge checks on every supported Python; the aggregator job `Lint (all versions)` is the single required status. +- **`tests-check.yml`** — `make gha-tests` on every supported Python; the aggregator job `Tests (all)` is the single required status. +- **`protocol-corpus-parity.yml`** — `tests/fixtures/protocol/` byte-identical with `mthds-js`, excluding `README.md` and the `*.fixture.ts` twins. It prefers the sibling's branch of the same name as the head and falls back to the sibling's `dev`. A `release/vX.Y.Z` head therefore reaches `dev` only when `mthds-js` holds no branch of that exact name: a release branch left on the sibling's origin after its own merge would be matched instead, and the two repos' version lines are independent but not disjoint. The job's `Resolve the sibling ref` step prints the ref it chose — read that rather than assuming `dev`. +- **`cla.yml`** — the CLA assistant, allowlisted through the `CLA_ALLOWLIST` repository variable. -Bumps version from `A.B.C` to `X.Y.Z`. +Nothing in CI compares the new version against the one already on `main`: `version-check.yml` checks equality with the branch name and no more, so a version that is not an increase passes the pull request and fails after the merge, at the PyPI upload. -### Changelog - - -``` +## Particulars -Report the PR URL back to the user, and remind them that **merging the PR into -`main` is what publishes** — `publish.yml` builds and pushes the package to PyPI -and cuts the GitHub release automatically. Nothing publishes until the PR is -merged. - -## Important details - -- The version follows semver: `MAJOR.MINOR.PATCH`. -- Always confirm the bump type with the user before making changes. -- If `make agent-check` fails, the release is blocked — help the user fix the - issues rather than skipping the checks. -- The CI gates a `release/vX.Y.Z` → `main` PR with: - - `version-check.yml` — the `pyproject.toml` version must match the - `release/vX.Y.Z` branch name. - - `changelog-check.yml` — `CHANGELOG.md` must contain a `## [vX.Y.Z] -` entry - for the new version. - - `package-check.yml` — `uv.lock` must be in sync with `pyproject.toml` - (`uv lock --locked`). - - `tests-check.yml` — the test matrix must pass on every supported Python - version (3.10 through 3.14). - - `lint-check.yml` — formatting, lint, and type checks (the same gates as - `make agent-check`). - - `guard-branches.yml` — only `release/vX.Y.Z` branches may target `main`. -- All checks must pass for the PR to be mergeable, so getting the changelog, - version, and lockfile right is critical. -- Pre-release versions are supported: a PEP 440 suffix (`a`, `b`, or `rc` - followed by a number, e.g. `0.6.0rc1`) makes `publish.yml` mark the GitHub - release as a pre-release. Use the branch name `release/v0.6.0rc1` to match. -- Today's date for the changelog entry: use the current date in `YYYY-MM-DD` - format. +- **No pre-release form reaches `main`.** `publish.yml` does detect a PEP 440 `a`, `b` or `rc` suffix and marks the GitHub Release a pre-release, which reads like permission — but nothing can carry such a version through: `guard-branches.yml` refuses a `release/v0.6.0rc1` head into `main` outright, and `version-check.yml` fails on the mismatch between a suffixed `pyproject.toml` version and the plain number in a `release/vX.Y.Z` branch name. Ship a plain `X.Y.Z`. The `pre-release/v*` base branch that `guard-branches.yml`'s `gate-release` job tolerates publishes nothing either, since `publish.yml` fires only on the push to `main`. +- **The changelog heading carries the `v`** — `## [vX.Y.Z] - YYYY-MM-DD`, which is exactly what `changelog-check.yml` greps for and what `publish.yml` slices the Release notes out of. No `[Unreleased]` heading is left behind; the next change re-creates one. +- **The fixture corpus is half of a pair.** A capture lands in `mthds-python` and `mthds-js` or in neither, so a release is also a statement that the sibling's `dev` carries the same bytes. That is what gate 3 and `protocol-corpus-parity.yml` are for. +- **A release arms no follow-ups on its own.** The ledger's `ledger.toml` declares no `release_followups` for this repo, so anything that must happen after this version ships is filed by hand alongside the release item in the play's step 3. +- **The back-merge is a merge commit.** `dev` carries a `Merge main into dev after the vX.Y.Z release` commit after each release rather than a fast-forward; `/ledger-land` makes it, and `CHANGELOG.md` is the one conflict it expects. diff --git a/CHANGELOG.md b/CHANGELOG.md index 97836cc..df0f40d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## [v0.15.0] - 2026-09-18 + +### Added + +- **`mthds.protocol.method_files` — the catalog serialization of a stored method's source**: `MethodFile` (`name` + `content`) and the pair `serialize_method_files` / `parse_method_files` between a list of files and the JSON `[{ name, content }]` string the hosted platform persists for a method's `.mthds` source and its custom PipeFunc `python`. The empty list serializes to `""`, the platform's "no source" sentinel, never to `"[]"`; blank-content entries are dropped in both directions; anything but the named array raises `PipelineRequestError`, including the three constants Python's `json` accepts and `JSON.parse` refuses. It mirrors the `mthds` npm package's `protocol/method_files.ts` edge for edge — blank is ECMAScript-blank, and the bytes are `JSON.stringify`'s down to both surrogate rules, an unpaired one escaped and an adjacent pair combined into the astral character Python alone can spell two ways — so a Python client reads a stored method's bundle from here rather than re-porting the platform's decoder. Python's own number model is kept out of the way too: integers decode as floats, the twin's single number type, so a literal longer than `sys.get_int_max_str_digits()` cannot raise a bare `ValueError` through the typed surface. One divergence from the twin is forced rather than chosen and is documented as such: Python's decoder recurses where `JSON.parse` iterates, so a deeply nested source is refused here and accepted there, at a depth that is an interpreter build constant. +- **`mthds.protocol.options` — the run-source argument surface:** predicates that decide which combinations of `pipe_code`, `mthds_contents`, the bundle encodings `files` / `bundle_b64`, `method_ref` and `method_id` may be sent together (`assert_exclusive_run_sources`, `has_bundle_payload`, `assert_method_ref_pairs_with_nothing`), plus the selector normalization `normalized_selector` / `run_selector_extensions`, which refuses a non-string selector. Nothing in this package calls them, so no client behaviour changes: `MthdsAPIClient` still forwards the keys inside `extra` without interpreting them. + +### Changed + +- **`ConceptAbstract` names a concept and nothing more, and `StuffAbstract` serializes `concept` as the ref string (Breaking)**: the protocol's concept model keeps `code`, `domain_code` and the `concept_ref` property — `description`, `structure_class_name` and `refines` are gone, since a concept's definition belongs to the library the method loads and never travels beside a stuff, and a runtime that needs one of them in memory declares it on its own subclass. `StuffAbstract` now carries a field serializer on `concept`, so every dump of a stuff emits `{"concept": ".", "content": …}` — the wire form the standard's [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/) requires — instead of leaving each runner to reduce the object itself. The dependency-contributed crate key `::.` is a subclass's to produce, by overriding `StuffAbstract.serialize_concept` under that name: the protocol model holds no declaring package address. The dump is one-way, which is the point rather than an oversight — a `StuffAbstract`, a `WorkingMemoryAbstract` or a `PipeOutputAbstract` can no longer be validated back from its own dump, since the definition such a parse would need is exactly what no longer travels. A consumer that used to re-validate a dump resolves the concept by ref through the library the method loads, the way the input side already does. Which of the two override points a runtime picks decides what this SDK's own output path emits: `DictRunResultExecute.from_pipe_output` reduces a stuff by `concept_ref`, so a `concept_ref` override reaches it and a `serialize_concept` override does not. + ## [v0.14.0] - 2026-09-06 ### Added diff --git a/README.md b/README.md index aaaa95f..4ad410f 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ pip install mthds ## What's in the box -- **The protocol** (`mthds.protocol`) — `MTHDSProtocol`, the five-route interface every runner implements (`execute`, `start`, `validate`, `models`, `version`); the wire models (`RunResult`, `ModelDeck`, `ValidationReport`, `VersionInfo`); and the domain shapes methods are built from (`concept`, `stuff`, `working_memory`, `pipe_output`, `pipeline_inputs`). +- **The protocol** (`mthds.protocol`) — `MTHDSProtocol`, the five-route interface every runner implements (`execute`, `start`, `validate`, `models`, `version`); the wire models (`RunResult`, `ModelDeck`, `ValidationReport`, `VersionInfo`); the domain shapes methods are built from (`concept`, `stuff`, `working_memory`, `pipe_output`, `pipeline_inputs`); and the catalog serialization of a stored method's source (`method_files`). - **Runners** (`mthds.runners`) — `MthdsAPIClient` (`mthds.runners.api.client`), the HTTP client for any MTHDS-Protocol server plus the hosted run-lifecycle (polling) extension; and `PipelexRunner` (`mthds.runners.pipelex.runner`), which shells out to a locally installed `pipelex` CLI. - **Package management** (`mthds.package`) — read, lock, and resolve `METHODS.toml` manifests. diff --git a/docs/runners.md b/docs/runners.md index 6d3aca1..da6c53e 100644 --- a/docs/runners.md +++ b/docs/runners.md @@ -6,7 +6,7 @@ The `mthds` package is the Python client for the open-source `pipelex-api` runne The protocol contract and its implementations live in separate packages: -- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). +- `mthds/protocol/` — the MTHDS Protocol itself: `protocol.py` (the `MTHDSProtocol` interface), `models.py` (the run/discovery wire models — `RunResultExecute`, `RunResultStart`, `ModelDeck`, `ValidationReport`, `VersionInfo`), `pipe_io_contracts.py` and `input_form.py` (the two validate artifacts the standard owns — see [The validate artifacts](#the-validate-artifacts-pipe_io_contracts-and-input_form) below), `method_files.py` (the catalog serialization of a stored method's source — see [The catalog serialization](#the-catalog-serialization-method_files) below), `options.py` (the run-source argument surface — see [Run sources: what may be combined](#run-sources-what-may-be-combined) below), `exceptions.py` (`PipelineRequestError`), and the protocol's domain shapes — `concept.py`, `stuff.py`, `working_memory.py`, `pipe_output.py`, `pipeline_inputs.py` (the abstract, non-Dict base models the protocol is defined in terms of). - `mthds/runners/` — every runner implementation, one subpackage per runner: - `api/` — the API runner: `client.py` (`MthdsAPIClient`, one file with its helpers), `models.py` (the Dict-serialized wire models — `DictConcept`, `DictStuffAbstract`, `DictWorkingMemoryAbstract`, `DictPipeOutputAbstract`, `DictRunResultExecute` — the runners' concrete JSON materialization of the protocol's domain shapes), `exceptions.py` (API auth + the protocol's 202-degrade error, `RunStillRunningError`). - `pipelex/runner.py` — `PipelexRunner`, the local runner that shells out to the `pipelex` CLI. @@ -64,6 +64,8 @@ async with MthdsAPIClient() as client: The Dict wire models are extension-open at every level, matching the protocol's OpenAPI spec (which declares `pipe_output` as an open object). Base fields are typed; anything more a runner returns rides `model_extra`. The hosted `pipelex-api` runner dumps its full `PipeOutput` — per-stuff `stuff_code` / `stuff_name`, pipe-output `graph_spec` / `tokens_usages` / `working_memory_raw` and assembly errors — and all of it is preserved. A stuff's `concept` accepts both wire forms: the reduced namespaced ref string (what this SDK's own serialization emits) or the full concept object (`DictConcept`, what the hosted runner dumps); `DictStuffAbstract.concept_ref` normalizes either form to the ref string. +On the producing side the protocol's own domain shapes say the same thing, and say it once: `ConceptAbstract` carries the reference alone (`code` + `domain_code`, with `concept_ref` joining them), and `StuffAbstract` serializes its `concept` field through that property, so every dump of a stuff emits `{"concept": ".", "content": …}` without a runner having to remember to reduce it. That is the standard's rule — a concept's definition (its description, its structure, what it refines, any runtime class name) belongs to the library the method loads and never travels beside a stuff, per the "Stuffs on the Wire" section of the [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/). A runtime that needs more of the definition in memory declares it on its own `ConceptAbstract` subclass, where it stays out of the wire form. Two consequences are worth naming. The crate key for a concept a dependency contributes is `::.`, and the protocol model holds no declaring package address, so producing that form is a subclass's business — it overrides `StuffAbstract.serialize_concept` under that name, since a second `field_serializer` for the field under any other name is a class-definition error and a serializer re-annotated onto the field is ignored silently. Which override point a runtime picks decides what this SDK's own output path emits, since `DictRunResultExecute.from_pipe_output` reduces a stuff by `concept_ref` rather than through the serializer: a `concept_ref` override reaches it, a `serialize_concept` override does not. And the dump is one-way: these shapes are strict, so a stuff, a working memory or a pipe output cannot be validated back from what it dumped, the definition such a parse would need being precisely what no longer travels. A consumer that re-validated a dump resolves the concept by ref through the library the method loads instead, as the input side already does; the `Dict` wire models above are the other way to read one back, since they accept either form. + ### The validate artifacts: `pipe_io_contracts` and `input_form` Two artifacts of the valid `/validate` report are owned by the standard since `mthds` v0.9.0 — [Pipe I/O Contracts](https://mthds.ai/latest/spec/pipe-io-contracts/) and the [Input-Form Descriptor](https://mthds.ai/latest/spec/input-form-descriptor/) — and this package types them: `mthds/protocol/pipe_io_contracts.py` and `mthds/protocol/input_form.py` mirror those pages exactly, snake_case slot for snake_case slot. Both are **recommended extension fields** of the report, not base fields: the protocol's base did not change, and how a caller asks a runner for the descriptor is implementation-defined (the hosted Pipelex API gates it behind its `views` request extension). They therefore arrive beside the report's typed base fields, and you narrow them by declaring them as typed fields on a model that extends the report — pydantic parses the maps and the discriminated union from the plain annotations, with no adapter machinery (this is exactly how `pipelex-sdk`'s report narrowing consumes them): @@ -110,6 +112,31 @@ What to know about the two modules: - **Parity with the TypeScript client is measured.** `tests/fixtures/protocol/` holds one real payload pair from the reference engine, committed byte-for-byte here and in `mthds-js`; the suite parses it strictly and asserts the dump equals the input. The fixture's README records the capture and the known engine drift the standard has since ruled on. - **The descriptor projects to a fill-in inputs template, client-side.** `mthds.protocol.inputs_template` turns one `PipeInputFormDescriptor` into the template somebody fills in and hands back — the compact and explicit shapes, as JSON or as TOML — so a client needs no server round trip to offer one for a method it does not have on disk. It is held to byte identity with the twin projection in the `mthds` npm package. See [docs/inputs-template.md](./inputs-template.md). +### The catalog serialization: `method_files` + +A stored method's source is persisted by the hosted platform as one string in the **catalog** form — the JSON `[{ name, content }]` array the webapp editor writes, one entry per bundle-relative file, for the method's `.mthds` source and again for its custom PipeFunc `python`. `mthds.protocol.method_files` is the one Python definition of that form: `MethodFile` (`name` + `content`, a closed strict shape) and the pair `serialize_method_files` / `parse_method_files` between a list of them and the stored string. It mirrors `protocol/method_files.ts` in the `mthds` npm package, so a client that fetches a stored method and wants its bundle back — an MCP server, your own code — reads it here instead of re-porting the platform's decoder. That is what the module is for, not yet a description of every consumer: `pipelex-sdk` still carries its own copy of this format, which disagrees with this one on blankness and on bytes, and moving it across is tracked as ledger item L-260912-6ef508. + +```python +from mthds.protocol.method_files import MethodFile, parse_method_files, serialize_method_files + +files = parse_method_files(stored_method["mthds"]) # -> list[MethodFile], in stored order; "" and "[]" both give [] +for method_file in files: + print(method_file.name, len(method_file.content)) + +stored = serialize_method_files([MethodFile(name="bundle.mthds", content=bundle_text)]) # -> the catalog string +``` + +What to know about it: + +- **It is the at-rest representation, not the run surface.** The catalog is an ordered, named array; a run request's `files` is an unordered path-to-text map. They are different shapes for different moments, and this module owns only the first. +- **No files is `""`, never `"[]"`.** The empty string is the platform's "no source" / "clear the field" sentinel, so the empty list serializes to it; the literal `"[]"` would be stored as a source. On the way back, a blank string, `None` and the empty array all parse to `[]`. +- **A blank file is not a file.** An entry whose content is empty or whitespace-only is dropped on both directions, so the round-trip is stable and the canonical form never carries an empty file. "Whitespace" here is what ECMAScript's `trim` strips — the byte-order mark included, U+001C–U+001F and U+0085 excluded — because the twin's verdict on a file must be this side's, and Python's `str.isspace` disagrees at both ends. One test derives that set from the Unicode categories rather than sampling it, so a code point cannot quietly go missing. Note that a zero-byte file is therefore not representable: an empty `__init__.py` does not survive a round trip. +- **The bytes are the twin's.** Serialization writes what `JSON.stringify` writes: compact separators, UTF-8 left as is, and only the JSON-mandated escapes. Surrogates take two rules, because Python addresses code points where JavaScript addresses UTF-16 code units: an unpaired surrogate is escaped rather than written raw — `ensure_ascii=False` alone would produce a string that cannot be encoded as UTF-8 at all — while two adjacent halves of a pair, which Python can hold as two code points and the twin cannot, are combined into the astral character the twin writes. The suite pins both against strings the twin printed. +- **Anything but the named array is a contract violation.** A source that is not JSON — a legacy raw bundle string included — a non-array value, an entry that is not an object with string `name` and string `content`, one of the three constants Python's `json` accepts and `JSON.parse` refuses (`NaN`, `Infinity`, `-Infinity`, an ignored extra member included) all raise `PipelineRequestError`. An entry's extra members are tolerated and stripped; a blank name is not a reason to drop an entry; duplicate names are kept, and since the platform refuses a method whose file names collide, uniqueness is the producer's obligation. An oversized integer is *not* a violation: Python's bounded `int()` conversion would refuse bytes the twin accepts, so integers are decoded as floats, which is the twin's own number model. +- **Deep nesting is the one place the two sides differ, and not by choice.** A source nested deeply enough to exhaust Python's recursive decoder raises `PipelineRequestError` rather than letting a `RecursionError` escape the typed surface — but the twin's `JSON.parse` is iterative and accepts any depth, so this side refuses what the twin accepts. The depth that trips it is an interpreter build constant: it rose by an order of magnitude in CPython 3.14 and differs between patch releases, so treat the guard as best-effort and never as a depth you can rely on. +- **A `name` is untrusted text.** Neither this module nor the twin checks that it is really bundle-relative, so an absolute or `..`-walking name round-trips untouched. Validate containment before writing a catalog to disk. +- **Where it departs from the platform's own decoder, it follows the twin.** The platform has no blank-source guard on its contents path at all, so a whitespace-only raw source fails downstream at the TOML parse; its own blankness is Python's `str.strip()`, which disagrees with the twin in both directions on an entry's content (a BOM-only file is content there and blank here, a U+001C–U+001F or U+0085 file the reverse); it drops blank-named entries; and it reads a non-array source — or a whole array containing one malformed entry — as a single legacy bundle, while silently dropping an entry whose content is not a string. This module and the TypeScript twin agree with each other on every one of those instead, so the three implementations never split two against one. **This package does not decode the legacy bare-bundle shape**, so a consumer reading one of the platform's very old plain-string rows must handle the `PipelineRequestError` itself — the example above assumes a catalog-form row. + ### Basic args vs extension args The abstract `MTHDSProtocol` interface carries the protocol's **basic** arguments only. Implementations may accept more (the protocol's extension policy), and the SDK passes any of them through: @@ -117,6 +144,25 @@ The abstract `MTHDSProtocol` interface carries the protocol's **basic** argument - Extension args never appear in this SDK — not even as convenience params. They ride the generic `extra` mapping on both `execute` and `start`: `client.start(pipe_code="answer", extra={"some_server_arg": True})` merges `some_server_arg` into the request body as a top-level property. The server you call defines and handles its own extension args; consult that server's API documentation for what it accepts. - Protocol args inside `extra` are rejected client-side with `PipelineRequestError` — pass them as named parameters. +### Run sources: what may be combined + +A run request has to name something to run, and several of the ways to name it exclude each other. That rule is an invariant of the request itself rather than of any one runner, so it lives in `mthds.protocol.options` for any Python client to enforce from one place instead of re-deriving it. **The definitions are there; no client enforces from them yet** — see "What is actually wired" below. The module is the twin of `mthds-js/src/protocol/options.ts`, whose two exported predicates it matches message for message; it is **not** a twin of either server, so a rejection here can read differently from the 422 it pre-empts (`pipelex-api` and the hosted platform each word these refusals their own way, and the platform checks the two exclusivity arms in the opposite order — `pipelex-api` has no two-encodings check at all). + +Three layers of argument meet in it: the protocol's own source (`pipe_code`, `mthds_contents`), the pipelex-api extensions that carry a whole method bundle (`files`, `bundle_b64`) or its published address (`method_ref`), and the hosted platform's catalog id (`method_id`). Only the first layer is a named parameter of `execute` / `start`; the others ride `extra` here, and are named parameters on a client that types its own stack's arguments. + +| Function | What it decides | +| --- | --- | +| `assert_exclusive_run_sources` | A bundle is self-contained, so it cannot be combined with `mthds_contents`, and `files` / `bundle_b64` are two encodings of one bundle. Keys off **presence** (`files={}` beside a zip is still two encodings), while `mthds_contents` counts only when non-empty. | +| `has_bundle_payload` | Whether a **runnable** bundle is present — an empty map or string carries no method, so it satisfies nothing. | +| `assert_method_ref_pairs_with_nothing` | A `method_ref` is a complete run source: exclusive with `mthds_contents`, with a bundle encoding, and with `method_id`. `pipe_code` beside it is legal (it overrides the fetched manifest's `main_pipe`), and inline source + `method_id` stays legal (the inline source runs, the id becomes run-history linkage). | +| `normalized_selector` / `run_selector_extensions` | The boundary normalization a selector goes through: a non-string is refused rather than silently dropped or forwarded to a server 422, and an absent or empty selector contributes nothing. | + +#### What is actually wired + +The table above describes the predicates, not a guarantee any client makes: **nothing in this package calls them.** `MthdsAPIClient.execute` / `start` check only that something to run was named — a pipe, inline contents, or any extension arg — and forward the keys inside `extra` without interpreting them. That is the client's documented contract: it serves any MTHDS-compliant runner, and which extension arg names a run source is that runner's to define. So an illegal combination sent through `extra` still reaches the server and comes back as its own 422, exactly as it did before this module existed. + +The predicates are there for a client that takes these arguments by name, which is where the combination is knowable before the request leaves. `PipelexAPIClient` in `pipelex-sdk` is the first such client; it still carries its own private copies and pins an older `mthds`, so replacing them waits on a release that ships this module. Which layer should enforce the extension rules — the generic client here, the client that names the arguments, or a Pipelex-scoped module beside this one — is still an open question, and the `pipe-selector` campaign will change this same surface after it is settled. + ## The durable run lifecycle (hosted API only) — lives in `pipelex-sdk` `start` (a protocol route) returns a `pipeline_run_id` only; turning that id into a result means polling the run to completion. That **durable run lifecycle is not part of the MTHDS Protocol** — it is a hosted-API extension, so it no longer lives in this package. It is exposed by `PipelexAPIClient` in `pipelex-sdk` (`get_run_status` / `get_run_result` / `wait_for_result` / `start_and_wait`), which builds on this protocol base. A bare runner serves no run store; the SDK detects that via the `version()` handshake and falls back to a blocking `execute`. diff --git a/mthds/protocol/concept.py b/mthds/protocol/concept.py index 692f08a..d1c71fd 100644 --- a/mthds/protocol/concept.py +++ b/mthds/protocol/concept.py @@ -7,14 +7,28 @@ class ConceptAbstract(BaseModel, ABC): + """A concept as a stuff names it — a reference, and nothing more. + + The standard puts a concept's definition — its description, its structure, + what it refines, and anything an implementation attaches to it such as the + name of a runtime class — in the library the method loads, never beside a + stuff (see the "Stuffs on the Wire" section of the + [CLI I/O contract](https://mthds.ai/latest/spec/cli-io-contract/)). A runtime + that needs more of the definition in memory declares it on its own subclass. + """ + model_config = ConfigDict(extra="forbid", strict=True) code: str domain_code: str - description: str - structure_class_name: str - refines: str | None = None @property def concept_ref(self) -> str: + """The domain-qualified reference, `.`. + + This is the crate key for a concept the method's own package declares and + for a native one. A concept a dependency contributes is keyed + `::.`, and this model holds no declaring + package address, so that form is a subclass's to produce. + """ return f"{self.domain_code}.{self.code}" diff --git a/mthds/protocol/method_files.py b/mthds/protocol/method_files.py new file mode 100644 index 0000000..306572f --- /dev/null +++ b/mthds/protocol/method_files.py @@ -0,0 +1,278 @@ +"""The canonical **catalog serialization** of a method's source files — mirrors `mthds-js`'s `protocol/method_files.ts`. + + catalog string ──parse_method_files──▶ list[MethodFile] (ordered, named) + list[MethodFile] ──serialize_method_files──▶ catalog string + +The catalog string is the JSON `[{ name, content }]` array form (the webapp-editor format) +that the hosted platform persists for a stored method's `.mthds` source and its custom +PipeFunc `python`. One canonical type plus one (de)serializer, so every Python consumer — +`pipelex-sdk`, an MCP server, a client that fetches a stored method and wants its bundle +back — agrees on the shape instead of each re-porting the platform's decoder and drifting. +The TypeScript twin exists for the same reason on its side; the two are held to the same +behaviour, edge for edge, and the test suite pins the Python side against bytes produced by +running the twin. + +This is the AT-REST catalog representation and is distinct from the run-surface `files` +map (an unordered path-to-text mapping on a run request): the catalog stores an ordered, +named ARRAY. The empty list is serialized as the empty string `""` — the platform's "no +source" / "clear the field" sentinel — never as the literal `"[]"`. + +The rules a port gets wrong while still round-tripping the happy case, all stated here: + +- **Blank means ECMAScript-blank.** A file carries no source when its content is empty or + whitespace-only, and "whitespace" is what `String.prototype.trim` strips — the closed set + in `_ECMASCRIPT_TRIM_CHARS` below — not what Python's `str.isspace` says. The two sets + disagree at both ends: Python strips U+001C to U+001F and U+0085, which the twin keeps as + content; the twin strips the byte-order mark U+FEFF, which Python keeps. A file the twin + drops must be a file this side drops, so the predicate is spelled out rather than borrowed + from `str.strip()`, and a test derives it from the Unicode categories rather than sampling it. +- **The bytes are `JSON.stringify`'s.** Compact separators, non-ASCII text left as UTF-8, and + only the characters JSON requires escaped. Python's `json.dumps` defaults to spaced + separators and backslash-u escapes for anything outside ASCII: same value, different bytes, + and an at-rest representation that hashes or compares differently on the two sides. +- **Surrogates are where Python's code points and the twin's UTF-16 code units disagree.** A + Python `str` addresses code points, so it can hold a lone surrogate *and* it can hold the two + halves of a pair as two adjacent code points; a JavaScript string addresses code units, so that + same sequence is one astral character to the twin. `ensure_ascii=False` writes either raw, and + for a lone surrogate the result cannot be encoded as UTF-8 at all — a break at the storage + boundary rather than at this typed surface. The twin's well-formed `JSON.stringify` escapes a + lone surrogate and emits a pair as the character it encodes, so `_coalesce_surrogate_pairs` + combines pairs before the dump and `_escape_lone_surrogates` escapes whatever is left after it. +- **`json.loads` accepts three constants `JSON.parse` refuses.** `NaN`, `Infinity` and + `-Infinity` are a Python extension to JSON. A catalog string carrying one anywhere parses here + and throws in the twin — and an ignored extra member is the case that bites, because no later + shape check ever looks there. `parse_constant` refuses all three, so the two sides do not + return opposite verdicts on those bytes. +- **Python's integer conversion is bounded and the twin's is not.** `json.loads` runs every + integer through `int()`, which since 3.11 refuses a literal longer than + `sys.get_int_max_str_digits()` with a bare `ValueError` — not a `JSONDecodeError` — so a long + enough integer inside an ignored extra member escaped this typed surface entirely while the twin + parsed the same bytes and stripped the member. JavaScript has one number type and no such limit, + so integers are decoded as floats here (`parse_int=float`): that is the twin's own model, it is + immune to the limit, and nothing kept is affected, because every number on this surface is + either discarded with its extra member or refused by the shape check. + +One divergence from the twin is forced rather than chosen, and it is the only one: `JSON.parse` is +iterative and accepts nesting of any depth, while Python's decoder recurses and raises +`RecursionError` past a threshold that is an interpreter build constant — it moved by an order of +magnitude in CPython 3.14, and it differs even between patch releases. That error is caught and +re-raised as a `PipelineRequestError` rather than left to escape this typed surface, so a source +nested deeply enough is refused here and accepted there. The guard is best-effort by nature: +the depth that trips it cannot be predicted, so the test discovers it rather than pinning it. + +Where this deliberately departs from the platform's own decoder (`method_source_to_contents` +and `parse_stored_files` in `pipelex-server`'s platform member), it follows the twin, so that +the three implementations do not split two against one: + +- The platform has no blank-source guard on the contents path at all — `method_source_to_contents` + goes straight to `json.loads` and hands back the raw string when that fails — so a + whitespace-only raw source becomes a whitespace bundle there and fails downstream at the TOML + parse; here, as in the twin, it is "no source". +- The platform's own blankness is Python's `str.strip()`, so it disagrees with the twin in both + directions on an entry's content: a BOM-only file is content to the platform and blank here, + while a file of U+001C to U+001F or U+0085 is blank to the platform and content here. Here, as + in the twin, blankness is ECMAScript's. +- The platform drops an entry whose `name` is blank; here, as in the twin, the name is not + inspected — only a blank `content` drops an entry. +- The platform treats a source that is not the file-array form as one legacy bare bundle, reads + a whole array as one such bundle as soon as a single entry is malformed, and drops an entry + whose `content` is not a string while keeping the rest. Here, as in the twin, every one of + those is a contract violation of this typed surface, and **this package does not decode the + legacy bare-bundle shape at all**: a consumer holding one of the platform's very old + plain-string rows must handle the `PipelineRequestError` itself, because the catalog + file-array is always the named-array form. + +Two obligations this surface does not discharge, which its consumers therefore own: + +- **A `name` is untrusted text, not a checked path.** Neither this module nor the twin validates + it, so `../../../etc/passwd`, an absolute path, a Windows path and the empty string all + round-trip untouched. Anything that materializes a catalog to disk must check containment + itself; this module only preserves what the catalog held. +- **Duplicate names are kept here and refused downstream.** The catalog form permits them, and + the platform rejects a method whose `.mthds` and `.py` file names collide, so uniqueness is + the producer's obligation rather than something a parse will tell it about. +""" + +from __future__ import annotations + +import json +import re +from typing import TYPE_CHECKING, Any, NoReturn, cast + +from pydantic import BaseModel, ConfigDict + +from mthds.protocol.exceptions import PipelineRequestError + +if TYPE_CHECKING: + from collections.abc import Sequence + + +class MethodFile(BaseModel): + """One named source file of a method bundle: a bundle-relative path and its text. + + A closed, strict shape — a consumer building one gets exactly `name` and `content`, both + strings. Leniency toward what the wire carries (extra members, which are tolerated and + stripped) belongs to `parse_method_files`, not to the type. + """ + + model_config = ConfigDict(extra="forbid", strict=True) + + name: str + """Bundle-relative path, e.g. `"funcs/price.py"` or `"bundle.mthds"`. + + Untrusted text, preserved exactly as the catalog held it: neither this type nor the twin + checks that it is in fact relative, so a consumer writing it to disk validates containment. + """ + + content: str + """The file's UTF-8 text content.""" + + +# The code points ECMAScript's `String.prototype.trim` strips — WhiteSpace (TAB, VT, FF, SP, +# NBSP, ZWNBSP and every Zs-category character) plus the four LineTerminators — spelled out +# so the blank predicate is the twin's, not `str.isspace`'s. Python's set is wider on the +# control side (it strips U+001C to U+001F and U+0085, which JavaScript keeps) and narrower at +# U+FEFF (which JavaScript strips); the tests pin every one of those disagreements, and one of +# them derives this set from the Unicode categories so a dropped code point cannot pass. +_ECMASCRIPT_TRIM_CHARS = ( + "\t\n\v\f\r \u00a0\u1680\u2000\u2001\u2002\u2003\u2004\u2005\u2006\u2007\u2008\u2009\u200a\u2028\u2029\u202f\u205f\u3000\ufeff" +) + + +def _is_blank(text: str) -> bool: + """A file carries no source when its content is empty or ECMAScript-whitespace-only.""" + return text.strip(_ECMASCRIPT_TRIM_CHARS) == "" + + +# A Python `str` addresses code points, so it can hold the two halves of a UTF-16 surrogate pair +# as two adjacent code points. A JavaScript string addresses code units, so that same sequence is +# one astral character to the twin, whose well-formed `JSON.stringify` (ES2019) writes it raw and +# escapes only a surrogate that is genuinely unpaired. Combining the pairs first is therefore what +# makes the two sides agree on a value only Python can spell two ways. +_SURROGATE_PAIR = re.compile(r"([\ud800-\udbff])([\udc00-\udfff])") + +# Whatever survives that pass cannot be part of an astral character, and +# `json.dumps(ensure_ascii=False)` writes it raw, producing a `str` that cannot be encoded as +# UTF-8 at all — so the twin's escape is reapplied after the dump. +_LONE_SURROGATE = re.compile(r"[\ud800-\udfff]") + + +def _coalesce_surrogate_pairs(text: str) -> str: + """Combine each adjacent high+low surrogate into the astral character it encodes, as the twin holds it. + + Left to right, so a lone high surrogate immediately before a well-formed pair stays lone — + which is what the twin does with the same code units. + """ + + def combine(match: re.Match[str]) -> str: + high, low = match.group(1), match.group(2) + return chr(0x10000 + ((ord(high) - 0xD800) << 10) + (ord(low) - 0xDC00)) + + return _SURROGATE_PAIR.sub(combine, text) + + +def _escape_lone_surrogates(dumped: str) -> str: + """Escape unpaired surrogates the way `JSON.stringify` does, leaving every other character raw.""" + return _LONE_SURROGATE.sub(lambda match: f"\\u{ord(match.group()):04x}", dumped) + + +def _refuse_json_constant(constant: str) -> NoReturn: + """Refuse the three constants `json.loads` accepts as an extension and `JSON.parse` rejects.""" + msg = f"Method file source is not valid JSON; expected a [{{ name, content }}] array (got `{constant}`)." + raise PipelineRequestError(msg) + + +def serialize_method_files(files: Sequence[MethodFile]) -> str: + """Serialize method files to the canonical catalog string. + + Blank-content entries are dropped (a zero-source file is not persisted), so the canonical + form never carries an empty file — which also means a list of nothing but blank files and an + empty list are the same call, both signalling "clear the stored source". An empty result + serializes to `""` — the platform's "no source" / "clear the field" signal — not `"[]"`. The + bytes are those of the twin's `JSON.stringify`: compact, UTF-8 left as is, only the + JSON-mandated escapes, and an unpaired surrogate escaped rather than written raw. + + Args: + files: The files, in the order they are to be stored. + + Returns: + The catalog string, or `""` when no file carries source. + """ + kept: list[dict[str, str]] = [ + {"name": _coalesce_surrogate_pairs(method_file.name), "content": _coalesce_surrogate_pairs(method_file.content)} + for method_file in files + if not _is_blank(method_file.content) + ] + if not kept: + return "" + return _escape_lone_surrogates(json.dumps(kept, ensure_ascii=False, separators=(",", ":"))) + + +def parse_method_files(source: str | None) -> list[MethodFile]: + """Parse the canonical catalog string back into method files. + + A blank source (`""` / whitespace / `None`) and an empty JSON array both yield `[]`. A JSON + `[{ name, content }]` array yields those files in order, with blank-content entries dropped + (mirroring serialization, so the round-trip is stable) and any extra member of an entry + ignored. Duplicate names are kept, and a blank name is not a reason to drop an entry. + + Anything else — a non-array JSON value, an array entry that is not an object with string + `name` and string `content`, unparseable text (a legacy raw bundle string included), or one of + the three constants `json.loads` accepts as an extension and `JSON.parse` refuses (`NaN`, + `Infinity`, `-Infinity`) — is a contract violation for this typed surface and raises + `PipelineRequestError`. A source that exhausts the decoder raises it too, but as a best-effort + guard rather than a shared rule: the twin's `JSON.parse` is iterative and accepts nesting this + side refuses, which is the module's one forced divergence from it. + + Args: + source: The stored catalog string, or `None` for a field that was never set. + + Returns: + The files the catalog holds, in stored order. + """ + if source is None or _is_blank(source): + return [] + + parsed: Any + try: + # `parse_int=float` is the twin's own number model — JavaScript has one number type — and + # it sidesteps Python's bounded `int()` conversion, whose bare `ValueError` would otherwise + # escape this surface. `ValueError` below subsumes `JSONDecodeError` and covers any other + # decoder refusal; `RecursionError` is not a `ValueError`, so it stays named. + # `PipelineRequestError` does not subclass `ValueError`, so the constant refusal above + # passes through carrying its own message — if that base ever changes, this clause has to + # re-raise it before wrapping, or that message is lost. + parsed = json.loads(source, parse_constant=_refuse_json_constant, parse_int=float) + except (ValueError, RecursionError) as exc: + msg = "Method file source is not valid JSON; expected a [{ name, content }] array." + raise PipelineRequestError(msg) from exc + + if not isinstance(parsed, list): + msg = "Method file source must be a JSON array of { name, content } entries." + raise PipelineRequestError(msg) + + files: list[MethodFile] = [] + for entry in cast("list[Any]", parsed): + method_file = _method_file_from_entry(entry) + if method_file is None: + msg = "Each method file entry must be an object with string `name` and string `content`." + raise PipelineRequestError(msg) + if not _is_blank(method_file.content): + files.append(method_file) + return files + + +def _method_file_from_entry(entry: Any) -> MethodFile | None: + """Read one array entry as a file, or `None` when it is not the `{ name, content }` shape. + + Only the two members are read and both must be strings; anything else the entry carries + is ignored, as the twin ignores it. + """ + if not isinstance(entry, dict): + return None + members = cast("dict[str, Any]", entry) + name = members.get("name") + content = members.get("content") + if not isinstance(name, str) or not isinstance(content, str): + return None + return MethodFile(name=name, content=content) diff --git a/mthds/protocol/options.py b/mthds/protocol/options.py new file mode 100644 index 0000000..008df7b --- /dev/null +++ b/mthds/protocol/options.py @@ -0,0 +1,242 @@ +"""The MTHDS Protocol's run-source argument surface — which combinations of the `execute` / +`start` arguments name something to run, and which are illegal. + +The protocol has no request *model*: a runner takes the request's basic arguments as named +parameters and serializes the wire body directly, merging any server-specific extension args +(`extra`) into the body as top-level properties. The Python expression of that argument surface +is therefore not a class but this set of predicates over the arguments themselves — each taking +plain values, so a caller passes them from wherever its own signature keeps them (a named +parameter, or a key it pulled out of `extra`). + +They travel with the request rather than with any one runner: which source combinations are +legal is an invariant of the request itself, so a Python client that builds one can enforce it +from this single definition instead of re-deriving it and drifting. **This module is that +definition, and nothing in the Python stack enforces from it yet.** `MthdsAPIClient` in this +package calls none of these predicates: it is a client for any MTHDS-compliant runner, so it +forwards the keys inside `extra` without interpreting them and leaves the addressed server to +answer for the combination. `PipelexAPIClient` in `pipelex-sdk` is the client that takes these +arguments by name and can pre-empt that refusal, and it still carries its own private copies — +it pins an older `mthds` and cannot import this module until one ships with it. + +Which layer should enforce the extension rules — the generic client, the client that names the +arguments, or a Pipelex-scoped module beside this one — is still open. These definitions are the +part that is settled. + +This module is the Python twin of `mthds-js/src/protocol/options.ts`, whose two exported +predicates it matches message for message. It is **not** a twin of either server, and a +rejection here can therefore read differently from the 422 it pre-empts: `pipelex-api` phrases +the bundle-versus-contents refusal its own way, and the platform validator tests the two +exclusivity arms in the opposite order, so one illegal body can draw a different sentence from +each. The JS twin is the binding parity; the servers are the inspiration. + +Three layers of argument meet here, and each predicate's docstring says which layer it is about: + +- **the protocol's own run source** — `pipe_code`, `mthds_contents`; +- **the pipelex-api extensions** that carry a whole method bundle (`files`, `bundle_b64`) or its + published address (`method_ref`); +- **the hosted platform's** catalog id (`method_id`). + +Only the first layer is a named parameter of `MTHDSProtocol.execute` / `start`; the other two +reach the wire through the generic `extra` passthrough, or as named parameters of a client that +types its own stack's arguments. All three are stated here on one argument: a rule split across +the packages that enforce it is a rule that drifts, which is the situation today and the one +this module exists to end. + +The `pipe-selector` campaign will change this surface (a `pipe_ref` beside `pipe_code`, and +which of the two may be combined with what). When it does, it changes it here. +""" + +from collections.abc import Mapping +from typing import Final + +from mthds.protocol.exceptions import PipelineRequestError + +RUN_ARG_FILES: Final[str] = "files" +"""Wire key of the pipelex-api `files` extension — a whole method bundle as a path-to-text map.""" + +RUN_ARG_BUNDLE_B64: Final[str] = "bundle_b64" +"""Wire key of the pipelex-api `bundle_b64` extension — the same bundle as a base64 zip.""" + +RUN_ARG_METHOD_REF: Final[str] = "method_ref" +"""Wire key of the pipelex-api `method_ref` extension — a published method's address.""" + +RUN_ARG_METHOD_ID: Final[str] = "method_id" +"""Wire key of the hosted platform's `method_id` run arg — a stored method's catalog id.""" + + +def normalized_selector(*, name: str, value: object) -> str | None: + """Normalize one method-selector argument at a client's boundary. + + A **non-string** value is refused rather than dropped or forwarded. A published client + validates its request-option types at its own boundary, so that one wrong value gets one + answer, rather than a bare truthiness check silently dropping the falsy wrong types (`0`, + `[]`) and forwarding the truthy ones (`123`, `["mt_1"]`) to a server `422`. This is a + deliberate DIVERGENCE from the JS lineage rather than parity with it: `nonEmptyString` in + `pipelex-sdk-js/src/client.ts` never throws, so it drops most wrong types silently and + forwards a `["mt_1"]` whose `length` happens to be non-zero. Refusing at the boundary is the + better behaviour, but the two sides do not agree on it yet. `value` is typed `object` rather + than `str | None` deliberately — this helper *is* the runtime boundary, and the callers it + guards against are the untyped ones a type checker never sees. + + An absent or **empty** value normalizes to `None`: an empty selector selects nothing, so it + is not sent and does not satisfy a client's "something to run" precondition. + + Args: + name: The argument's name, as it appears in the error message and on the wire. + value: The caller-supplied value, of any type. + + Returns: + The selector string, or None when it is absent or empty. + + Raises: + PipelineRequestError: If the value is present and is not a string. + """ + if value is None: + return None + if not isinstance(value, str): + msg = f"{name} must be a string, received {type(value).__name__}." + raise PipelineRequestError(msg) + return value or None + + +def run_selector_extensions(*, method_ref: object = None, method_id: object = None) -> dict[str, str]: + """Normalize the method selectors a client names into the extension mapping they ride to the + wire. + + The layering seam: `method_ref` (layer 2 — the Pipelex API's run source, resolved by the + runner) and `method_id` (layer 3 — the hosted platform's run arg) are named parameters on + the client that types its own stack's arguments, and each reaches the wire as a top-level + body property through the protocol client's generic extension mechanism, which merges it + without knowing what it means. + + Both go through `normalized_selector`, so a non-string is refused and an absent or empty + value contributes nothing. Rejecting the caller's own reserved argument names inside `extra` + is the calling client's business — it owns that list — and merging is left to the caller too, + so this returns only what the selectors contribute. + + Args: + method_ref: The published method's address, or None. + method_id: The hosted catalog id, or None. + + Returns: + The selector contribution to the extension mapping — empty when neither is present. + + Raises: + PipelineRequestError: If a selector is present and is not a string. + """ + extensions: dict[str, str] = {} + selected_method_ref = normalized_selector(name=RUN_ARG_METHOD_REF, value=method_ref) + if selected_method_ref is not None: + extensions[RUN_ARG_METHOD_REF] = selected_method_ref + selected_method_id = normalized_selector(name=RUN_ARG_METHOD_ID, value=method_id) + if selected_method_id is not None: + extensions[RUN_ARG_METHOD_ID] = selected_method_id + return extensions + + +def has_bundle_payload(*, files: Mapping[str, str] | None = None, bundle_b64: str | None = None) -> bool: + """Does the request carry a method bundle (the pipelex-api `files` / `bundle_b64` extension)? + + A bundle satisfies a client's "something to run" precondition on its own — it carries its + own `.mthds`, so neither `pipe_code` nor `mthds_contents` is required alongside it. Unlike + `assert_exclusive_run_sources`, this keys off a RUNNABLE payload: an empty map or string + carries no method, so it satisfies nothing (and must not be sent — the runner rejects a + zero-file bundle). + + Args: + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + + Returns: + True when either encoding carries a method. + """ + return bool(files) or bool(bundle_b64) + + +def assert_exclusive_run_sources( + *, + mthds_contents: list[str] | None = None, + files: Mapping[str, str] | None = None, + bundle_b64: str | None = None, +) -> None: + """Enforce the run-source exclusivity contract shared by every client: a method bundle is + self-contained (`files` / `bundle_b64` carry their own `.mthds`), so it cannot be combined + with `mthds_contents`, and `files` / `bundle_b64` are two encodings of one bundle. + + Exclusivity keys off PRESENCE, not emptiness — a caller who supplies `files={}` alongside + `bundle_b64` still expressed two encodings — while `mthds_contents` counts only when + non-empty (an empty list is "no contents"). Both servers count these values the same way, + but the wording is the `mthds-js` twin's, byte for byte, and neither server's. + + Args: + mthds_contents: Inline MTHDS bundle contents, or None. + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + + Raises: + PipelineRequestError: If two run sources that exclude each other are both present. + """ + has_files = files is not None + has_zip = bundle_b64 is not None + has_contents = bool(mthds_contents) + if has_files and has_zip: + msg = "files and bundle_b64 are two encodings of the same bundle and are mutually exclusive; provide one." + raise PipelineRequestError(msg) + if (has_files or has_zip) and has_contents: + msg = "A method bundle (files/bundle_b64) is self-contained; it cannot be combined with mthds_contents." + raise PipelineRequestError(msg) + + +def assert_method_ref_pairs_with_nothing( + *, + method_ref: object = None, + mthds_contents: list[str] | None = None, + files: Mapping[str, str] | None = None, + bundle_b64: str | None = None, + method_id: object = None, +) -> None: + """Enforce the run routes' `method_ref` exclusivity, so an illegal pairing fails before + anything hits the wire. + + A `method_ref` is a complete run source (the fetched package carries its `.mthds` and its + entry pipe), so it pairs with NOTHING: not with inline `mthds_contents`, not with a bundle + encoding, and not with the hosted `method_id` — an address run has its own provenance and + needs no linkage id. + + The one documented run-route exception is deliberately NOT here: inline source + `method_id` + stays legal (the inline source runs; the id demotes to run-history linkage). `pipe_code` + beside a `method_ref` is legal too — it overrides the fetched manifest's `main_pipe`. + `mthds_contents` counts when non-empty and a bundle encoding counts when the key is present, + as on both servers. A selector counts only when non-empty, which is not the servers' rule: + they declare `min_length=1`, so an empty selector is a `422` there rather than absent. The + messages are those of the private `assertMethodRefPairsWithNothing` in + `pipelex-sdk-js/src/client.ts`, byte for byte; `mthds-js` exports no counterpart. + + Args: + method_ref: The published method's address, or None — when absent or empty, no pairing + is checked, though both selectors are still refused if present and not a string. + mthds_contents: Inline MTHDS bundle contents, or None. + files: The bundle as a path-to-text map, or None. + bundle_b64: The bundle as a base64-encoded zip, or None. + method_id: The hosted catalog id, or None. + + Raises: + PipelineRequestError: If a `method_ref` is paired with any other run source, or if a + selector is present and is not a string. + """ + selected_method_ref = normalized_selector(name=RUN_ARG_METHOD_REF, value=method_ref) + selected_method_id = normalized_selector(name=RUN_ARG_METHOD_ID, value=method_id) + if selected_method_ref is None: + return + if mthds_contents: + msg = "method_ref and inline mthds_contents are mutually exclusive; send one or the other." + raise PipelineRequestError(msg) + if files is not None or bundle_b64 is not None: + msg = "method_ref and a method bundle (bundle_b64 / files) are mutually exclusive; send one or the other." + raise PipelineRequestError(msg) + if selected_method_id is not None: + msg = ( + "method_ref and method_id are mutually exclusive: an address run carries its own provenance " + "and takes no run-history linkage id. Send exactly one method selector." + ) + raise PipelineRequestError(msg) diff --git a/mthds/protocol/stuff.py b/mthds/protocol/stuff.py index 350fd8c..6adfdb9 100644 --- a/mthds/protocol/stuff.py +++ b/mthds/protocol/stuff.py @@ -1,7 +1,7 @@ from abc import ABC from typing import Any, Generic, TypeVar -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, field_serializer from mthds.protocol.concept import ConceptType @@ -17,6 +17,34 @@ class StuffAbstract(BaseModel, ABC, Generic[ConceptType, StuffContentType]): concept: ConceptType content: StuffContentType + @field_serializer("concept") + def serialize_concept(self, concept: ConceptType) -> str: + """Emit the concept as its reference string. + + A stuff names its concept and never carries its definition, so every + dump of a stuff — whatever a runtime keeps on its own concept subclass — + is the standard's wire form. + + The dump is therefore one-way: these models are strict, so a stuff, a + working memory or a pipe output cannot be validated back from its own + dump. That is the point rather than an oversight — the definition the + parse would need is not on the wire, and a reader resolves the ref + through the concept library the method loads, the way the input side + already does. + + A subclass that emits something else — the `::.` + crate key of a dependency-contributed concept, say — has two ways in, and + they do not reach the same places. Overriding `concept_ref` on the concept + reaches every reduction, this serializer included. Overriding this method + reaches a dump and nothing else: it must keep **this name**, decorated + again or as a plain method, since a second `field_serializer` for + `concept` under another name raises pydantic's `multiple-field-serializers` + error at class definition, while re-annotating the field with a serializer + or putting a `model_serializer` on the concept subclass is ignored in + favour of this one with no diagnostic at all. + """ + return concept.concept_ref + class StuffContentAbstract(BaseModel, ABC): pass diff --git a/mthds/runners/api/client.py b/mthds/runners/api/client.py index 66e16cb..df3e97f 100644 --- a/mthds/runners/api/client.py +++ b/mthds/runners/api/client.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import TYPE_CHECKING, Any, ClassVar, Self, cast +from typing import TYPE_CHECKING, Any, ClassVar, Literal, Self, cast from urllib.parse import quote import httpx @@ -161,12 +161,12 @@ async def execute( Complete execution results including run state and output Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a + protocol arg. RunStillRunningError: If the server answers 202 (the protocol's optional async degrade) — the run continues server-side; resume by `run_id`. """ - if not pipe_code and not mthds_contents and not extra: - msg = "Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API execute." - raise PipelineRequestError(msg) + _assert_run_sources(pipe_code=pipe_code, mthds_contents=mthds_contents, extra=extra, route="execute") body = _build_run_body( pipe_code=pipe_code, @@ -214,10 +214,12 @@ async def start( RunResultStart — the authoritative server-generated `pipeline_run_id` (no output yet). On a hosted deployment the id is durable — poll the durable run lifecycle (a hosted extension, exposed by `pipelex-sdk`). + + Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a + protocol arg. """ - if not pipe_code and not mthds_contents and not extra: - msg = "Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API start." - raise PipelineRequestError(msg) + _assert_run_sources(pipe_code=pipe_code, mthds_contents=mthds_contents, extra=extra, route="start") body = _build_run_body( pipe_code=pipe_code, @@ -388,6 +390,36 @@ def _raise_if_execute_degraded(self, response: httpx.Response) -> None: _VALIDATION_RESULT_ADAPTER: TypeAdapter[ValidationResult] = TypeAdapter(ValidationResult) +def _assert_run_sources( + *, pipe_code: str | None, mthds_contents: list[str] | None, extra: dict[str, Any] | None, route: Literal["execute", "start"] +) -> None: + """Guard the run-source arguments of one `/execute` | `/start` call, before any body is built. + + One check: something to run must have been named — a pipe, inline contents, or a + server-specific extension arg. Which extension arg names a run source, and which of them + exclude each other, is the server's own contract: this client serves any MTHDS-compliant + runner, so it counts the keys inside `extra` without interpreting them, and the runner + answers for the combination it was sent. A client that takes those arguments by name can + pre-empt the refusal from the shared definitions in `mthds.protocol.options`. + + The extension mapping is built here rather than only in `_build_run_body` so that a protocol + arg smuggled through `extra` is refused at this boundary, before any body exists. + + Args: + pipe_code: The pipe selector, or None. + mthds_contents: Inline MTHDS bundle contents, or None. + extra: Server-specific extension args from the caller, or None. + route: The protocol route being called, for the error message (`execute` / `start`). + + Raises: + PipelineRequestError: If nothing to run was named, or if `extra` carries a protocol arg. + """ + extensions = _build_extensions(extra) + if not pipe_code and not mthds_contents and not extensions: + msg = f"Either pipe_code, mthds_contents or a server-specific extension arg (extra) must be provided to the API {route}." + raise PipelineRequestError(msg) + + def _build_run_body( *, pipe_code: str | None, diff --git a/mthds/runners/api/models.py b/mthds/runners/api/models.py index d12832f..17000cb 100644 --- a/mthds/runners/api/models.py +++ b/mthds/runners/api/models.py @@ -113,6 +113,9 @@ def _serialize_working_memory(cls, working_memory: WorkingMemoryAbstract[StuffTy # Convert each Stuff -> DictStuff by dumping only the content for stuff_name, stuff in working_memory.root.items(): + # The concept is reduced by `concept_ref`, which is what the protocol's own field + # serializer delegates to. A runtime that overrides `serialize_concept` instead is + # not followed here: which override point this path should honour is L-260918-5f09d8. dict_stuff = cls._dict_stuff_class( concept=stuff.concept.concept_ref, content=stuff.content.model_dump(serialize_as_any=True), diff --git a/pyproject.toml b/pyproject.toml index 9194e6c..bcfb4da 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "mthds" -version = "0.14.0" +version = "0.15.0" description = "The Python interface for methods — base structures for structured outputs and the base runner for executing methods via API." authors = [{ name = "Evotis S.A.S.", email = "oss@pipelex.com" }] maintainers = [{ name = "Pipelex staff", email = "oss@pipelex.com" }] @@ -178,6 +178,13 @@ strictParameterNoneValue = true strictSetInference = true typeCheckingMode = "strict" +# Scoped once here rather than as a per-line ignore in each suite: a test may legitimately pin a +# module-private name whose exact value IS the contract, such as the ECMAScript trim set the +# catalog serialization's blank predicate is built from. Everything else stays strict. +[[tool.pyright.executionEnvironments]] +root = "tests" +reportPrivateUsage = "none" + [tool.ruff] exclude = [ ".cursor", @@ -326,6 +333,7 @@ convention = "google" "tests/**/*.py" = [ "implicit-namespace-package", # Allow test files to not have __init__.py in their directories (avoids namespace collisions) "float-equality-comparison", # Tests assert exact float literals that round-trip exactly; `pytest.approx` would only add noise + "import-private-name", # A test may pin a module-private constant whose exact value IS the contract, e.g. the ECMAScript trim set ] "examples/**/*.py" = [ "implicit-namespace-package", # Runnable demo scripts, not an importable package diff --git a/tests/unit/test_api_runner_models.py b/tests/unit/test_api_runner_models.py index f56b6d5..b2b989e 100644 --- a/tests/unit/test_api_runner_models.py +++ b/tests/unit/test_api_runner_models.py @@ -2,39 +2,44 @@ from typing import Any -from pydantic import BaseModel - +from mthds.protocol.concept import ConceptAbstract +from mthds.protocol.pipe_output import PipeOutputAbstract +from mthds.protocol.stuff import StuffAbstract, StuffContentAbstract +from mthds.protocol.working_memory import WorkingMemoryAbstract from mthds.runners.api.models import MAIN_STUFF_NAME, DictRunResultExecute -class _StubContent(BaseModel): +class _StubContent(StuffContentAbstract): value: str -class _StubConcept(BaseModel): - concept_ref: str +class _StubConcept(ConceptAbstract): + pass -class _StubStuff(BaseModel): - concept: _StubConcept - content: _StubContent +class _StubStuff(StuffAbstract[_StubConcept, _StubContent]): + pass -class _StubWorkingMemory(BaseModel): - root: dict[str, _StubStuff] - aliases: dict[str, str] +class _StubWorkingMemory(WorkingMemoryAbstract[_StubStuff]): + pass -class _StubPipeOutput(BaseModel): - working_memory: _StubWorkingMemory - pipeline_run_id: str +class _StubPipeOutput(PipeOutputAbstract[_StubWorkingMemory]): + pass def _pipe_output_with_run_id(run_id: str) -> Any: """A minimal pipe-output stub carrying a non-empty pipeline_run_id.""" return _StubPipeOutput( working_memory=_StubWorkingMemory( - root={"main": _StubStuff(concept=_StubConcept(concept_ref="answer.Answer"), content=_StubContent(value="42"))}, + root={ + "main": _StubStuff( + stuff_code="stuff_1", + concept=_StubConcept(code="Answer", domain_code="answer"), + content=_StubContent(value="42"), + ) + }, aliases={MAIN_STUFF_NAME: "main"}, ), pipeline_run_id=run_id, @@ -77,3 +82,11 @@ def test_main_stuff_name_extension_passes_through(self) -> None: assert result.model_extra is not None assert result.model_extra["main_stuff_name"] == "main" + + def test_concept_is_reduced_to_its_ref_string(self) -> None: + """The wire model carries the concept as `.` — the reduction this path + performs is `concept_ref`, the same property the protocol's field serializer delegates to. + """ + result = DictRunResultExecute.from_pipe_output(pipe_output=_pipe_output_with_run_id("run_1")) + + assert result.pipe_output.working_memory.root["main"].concept == "answer.Answer" diff --git a/tests/unit/test_api_runner_protocol.py b/tests/unit/test_api_runner_protocol.py index af5c591..d41d5f5 100644 --- a/tests/unit/test_api_runner_protocol.py +++ b/tests/unit/test_api_runner_protocol.py @@ -2,6 +2,7 @@ import asyncio import inspect +from typing import Any import httpx import pytest @@ -11,6 +12,7 @@ from mthds.protocol.models import InvalidValidationReport, ModelCategory, ModelDeck, ValidationReport, VersionInfo from mthds.protocol.protocol import MTHDSProtocol from mthds.runners.api.client import MthdsAPIClient +from tests.unit.test_data import ExecuteWireResponses _BASE_URL = "http://localhost:8081" @@ -198,3 +200,55 @@ def test_version_parses_handshake(self, mocker: MockerFixture) -> None: assert info.runner_version == "0.3.0" # Implementation identification passes through as extensions, never named by the SDK. assert info.model_extra == {"some_vendor_name": "vendor-runner"} + + # ── execute / start run sources ─────────────────────────────── + + @pytest.mark.parametrize( + ("route", "wire_response"), + [("execute", ExecuteWireResponses.REDUCED), ("start", {"pipeline_run_id": "run_7f3a"})], + ) + def test_run_requires_something_to_run(self, route: str, wire_response: dict[str, Any], mocker: MockerFixture) -> None: + """Neither a pipe, nor contents, nor an extension arg: the call is refused before any request. + + The transport stub answers with a well-formed body on purpose. Were the guard removed, + the call would run to completion and this test would report the guard's absence — rather + than a parse error from a stub that could never have produced a result anyway. + """ + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=wire_response))) + with pytest.raises(PipelineRequestError, match=f"must be provided to the API {route}"): + asyncio.run(getattr(client, route)()) + send_mock.assert_not_called() + + @pytest.mark.parametrize( + ("route", "wire_response"), + [("execute", ExecuteWireResponses.REDUCED), ("start", {"pipeline_run_id": "run_7f3a"})], + ) + def test_run_forwards_extension_args_without_interpreting_them(self, route: str, wire_response: dict[str, Any], mocker: MockerFixture) -> None: + """`extra` is the server's to define, so this client forwards its keys without judging them. + + The combination sent here — inline contents beside both bundle encodings at once — is + illegal on `pipelex-api`, which is the runner that defines `files` and `bundle_b64`. They + are that runner's extension args and not the protocol's, and `MthdsAPIClient` is + documented as a client for any MTHDS-compliant runner, so it is the addressed server that + answers for the combination. Pre-empting the refusal is the business of a client that + takes those arguments by name, from the shared definitions in `mthds.protocol.options`. + """ + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=wire_response))) + + asyncio.run(getattr(client, route)(mthds_contents=['domain = "answer"'], extra={"files": {}, "bundle_b64": "UEsDBA=="})) + + sent = send_mock.call_args.kwargs["content"].decode("utf-8") + assert '"mthds_contents":["domain = \\"answer\\""]' in sent + assert '"files":{}' in sent + assert '"bundle_b64":"UEsDBA=="' in sent + + def test_execute_accepts_a_bundle_as_the_only_run_source(self, mocker: MockerFixture) -> None: + """A bundle in `extra` satisfies the precondition and merges into the body as a top-level property.""" + client = self._client() + send_mock = mocker.patch.object(client, "_send", mocker.AsyncMock(return_value=_response(200, json=ExecuteWireResponses.REDUCED))) + + asyncio.run(client.execute(extra={"files": {"main.mthds": 'domain = "answer"'}})) + sent = send_mock.call_args.kwargs["content"].decode("utf-8") + assert '"files":{"main.mthds":"domain = \\"answer\\""}' in sent diff --git a/tests/unit/test_config_dialect.py b/tests/unit/test_config_dialect.py index 2dd9079..e57509b 100644 --- a/tests/unit/test_config_dialect.py +++ b/tests/unit/test_config_dialect.py @@ -14,7 +14,7 @@ # The dialect engine is deliberately private (an implementation detail of the config surface); # this suite pins the dialect itself, so it reaches the private names on purpose. -from mthds.config import _parse_dotenv, _serialize_dotenv # ruff: ignore[import-private-name] # pyright: ignore[reportPrivateUsage] +from mthds.config import _parse_dotenv, _serialize_dotenv # pyright: ignore[reportPrivateUsage] _FIXTURE_PATH = Path(__file__).resolve().parent.parent / "fixtures" / "config_dialect_cases.json" diff --git a/tests/unit/test_data.py b/tests/unit/test_data.py index 158c264..c69306d 100644 --- a/tests/unit/test_data.py +++ b/tests/unit/test_data.py @@ -590,3 +590,134 @@ class CompactSlotCases: item_count=2, item=ObjectItem(concept_ref="probe.Gadget", required=True, fields=[TextField(name="label", required=True)]), ) + + +class MethodFileCases: + """Vectors for the catalog serialization, pinned against the TypeScript twin. + + `TWIN_SERIALIZED` is the exact string `serializeMethodFiles(TWIN_FILES)` printed when run from + `mthds-js/src/protocol/method_files.ts` (2026-09-12, node 24): compact separators, UTF-8 left as + is, the double quote, the backslash and the LF and TAB it happens to contain escaped and nothing + else. `BLANK_PREDICATE` is what the same run said of each code point — which single-character + contents `serializeMethodFiles` dropped and which `parseMethodFiles` read as "no source" — so + every disagreement between ECMAScript's `trim` and Python's `str.isspace` is stated as a case + rather than assumed away. It samples the set; `test_the_trim_set_is_ecmascripts_whole_trim_set` + is what actually holds the set complete, by deriving it. + """ + + TWIN_FILES: ClassVar[list[dict[str, str]]] = [ + { + "name": "bundle.mthds", + "content": 'domain = "x"\n[concept.\u00c9t\u00e9]\n' + 'description = "caf\u00e9 \u2014 \u00fcn\u00efc\u00f6d\u00e9 \u65e5\u672c \U0001f642"\n', + }, + {"name": "funcs/price.py", "content": 'def price():\n\treturn "a\\\\b" + \'\\u00e9\' + "/" + "" + "&"\n'}, + ] + TWIN_SERIALIZED: ClassVar[str] = ( + '[{"name":"bundle.mthds","content":"domain = \\"x\\"\\n[concept.\u00c9t\u00e9]\\n' + 'description = \\"caf\u00e9 \u2014 \u00fcn\u00efc\u00f6d\u00e9 \u65e5\u672c \U0001f642\\"\\n"},' + '{"name":"funcs/price.py","content":"def price():\\n\\treturn \\"a\\\\\\\\b\\" + \'\\\\u00e9\' + \\"/\\" + \\"\\" + \\"&\\"\\n"}]' + ) + + # (topic, single-character content, whether the twin treats it as blank) + BLANK_PREDICATE: ClassVar[list[tuple[str, str, bool]]] = [ + ("U+0009 TAB", "\t", True), + ("U+000A LF", "\n", True), + ("U+000B VT", "\v", True), + ("U+000C FF", "\f", True), + ("U+000D CR", "\r", True), + ("U+0020 SPACE", " ", True), + ("U+00A0 NBSP", "\u00a0", True), + ("U+1680 OGHAM SPACE MARK", "\u1680", True), + ("U+2000 EN QUAD", "\u2000", True), + ("U+200A HAIR SPACE", "\u200a", True), + ("U+2028 LINE SEPARATOR", "\u2028", True), + ("U+2029 PARAGRAPH SEPARATOR", "\u2029", True), + ("U+202F NARROW NBSP", "\u202f", True), + ("U+205F MEDIUM MATHEMATICAL SPACE", "\u205f", True), + ("U+3000 IDEOGRAPHIC SPACE", "\u3000", True), + ("U+FEFF BOM — blank to the twin, not to str.isspace", "\ufeff", True), + ("U+001C FILE SEPARATOR — content to the twin, whitespace to str.isspace", "\x1c", False), + ("U+001D GROUP SEPARATOR — content to the twin, whitespace to str.isspace", "\x1d", False), + ("U+001E RECORD SEPARATOR — content to the twin, whitespace to str.isspace", "\x1e", False), + ("U+001F UNIT SEPARATOR — content to the twin, whitespace to str.isspace", "\x1f", False), + ("U+0085 NEXT LINE — content to the twin, whitespace to str.isspace", "\x85", False), + ("U+200B ZERO WIDTH SPACE — content on both sides", "\u200b", False), + ("U+180E MONGOLIAN VOWEL SEPARATOR — content on both sides", "\u180e", False), + ] + + # (topic, source) — every one read by the twin as "no files" + NO_FILES_SOURCES: ClassVar[list[tuple[str, str | None]]] = [ + ("the empty string", ""), + ("whitespace only", " "), + ("a mix of every whitespace kind, the BOM included", " \n\t\r \ufeff "), + ("None, a field never set", None), + ("the empty JSON array", "[]"), + ("the empty JSON array with trailing whitespace", "[] "), + ("the empty JSON array with leading whitespace", " []"), + ] + + # (topic, source) — every one refused by the twin with PipelineRequestError + CONTRACT_VIOLATIONS: ClassVar[list[tuple[str, str]]] = [ + ("raw bundle text is the legacy shape, not the catalog array", "domain = 'x'"), + ("a BOM before the array is not JSON on either side", "\ufeff[]"), + ("a JSON object, even one shaped like an entry", '{"name":"a.py","content":"x"}'), + ("JSON null", "null"), + ("a JSON string", '"abc"'), + ("a JSON number", "1"), + ("a bare NaN, refused by the decoder now and by the array check regardless", "NaN"), + ("an entry missing content", '[{"name":"a.py"}]'), + ("an entry that is a bare string", '["a.py"]'), + ("an entry that is null", "[null]"), + ("an entry that is an array", "[[]]"), + ("an entry whose name is not a string", '[{"name":1,"content":"x"}]'), + ("an entry whose content is not a string", '[{"name":"a","content":1}]'), + ("an entry whose content is null", '[{"name":"a","content":null}]'), + ("one good entry does not excuse a bad one", '[{"name":"a","content":"x"},{"name":"b"}]'), + # `json.loads` accepts these three as a Python extension and the twin's `JSON.parse` + # throws on every one. A bare `NaN` above is caught by the array check whatever the + # decoder does; inside an ignored member nothing downstream would ever look, so these + # are the cases that actually pin `parse_constant`. + ("NaN inside an ignored extra member", '[{"name":"a","content":"x","extra":NaN}]'), + ("Infinity inside an ignored extra member", '[{"name":"a","content":"x","extra":Infinity}]'), + ("-Infinity nested in an ignored member's array", '[{"name":"a","content":"x","extra":[1,-Infinity]}]'), + ("Infinity as a whole entry", "[Infinity]"), + ] + + # (topic, content, the exact string the twin's `JSON.stringify` printed for that one file) + # `json.dumps(ensure_ascii=False)` writes an unpaired surrogate raw, which is not UTF-8 + # encodable at all, so these fail at the storage boundary rather than at the typed surface. + LONE_SURROGATES: ClassVar[list[tuple[str, str, str]]] = [ + ("a lone high surrogate", "\ud800", '[{"name":"f","content":"\\ud800"}]'), + ("a lone low surrogate", "\udfff", '[{"name":"f","content":"\\udfff"}]'), + ("a lone surrogate between text", "a\ud800b", '[{"name":"f","content":"a\\ud800b"}]'), + ("an astral character stays raw, being one code point", "\U0001f642", '[{"name":"f","content":"\U0001f642"}]'), + ] + + # (topic, content, the exact string the twin's `JSON.stringify` printed for that one file) + # A Python `str` addresses code points and can hold the two halves of a pair as two of them, + # where the twin — addressing UTF-16 code units — holds one astral character and writes it + # raw. The pair is combined before the dump, so these are the twin's bytes and not two + # escapes. Built with `chr` so no source encoding can quietly collapse a pair into one code + # point, which is exactly the mistake that hides this case. + SURROGATE_PAIRS: ClassVar[list[tuple[str, str, str]]] = [ + ("a pair is the astral character it encodes", chr(0xD83D) + chr(0xDE42), '[{"name":"f","content":"\U0001f642"}]'), + ("the lowest pair", chr(0xD800) + chr(0xDC00), '[{"name":"f","content":"\U00010000"}]'), + ("the highest pair", chr(0xDBFF) + chr(0xDFFF), '[{"name":"f","content":"\U0010ffff"}]'), + ( + "a lone high before a pair stays lone, the pair still combines", + chr(0xD800) + chr(0xD800) + chr(0xDC00), + '[{"name":"f","content":"\\ud800\U00010000"}]', + ), + ("a low before a high is two lone surrogates", chr(0xDC00) + chr(0xD800), '[{"name":"f","content":"\\udc00\\ud800"}]'), + ( + "a pair followed by a lone low", + chr(0xD83D) + chr(0xDE42) + chr(0xDFFF), + '[{"name":"f","content":"\U0001f642\\udfff"}]', + ), + ] + + # A JSON integer long enough to exceed `sys.get_int_max_str_digits()` makes Python's decoder + # raise a bare `ValueError` from `int()` — not a `JSONDecodeError` — where the twin, having one + # number type and no such limit, parses the same bytes. The limit is 4300 digits by default. + OVERSIZED_INTEGER_DIGITS: ClassVar[int] = 4301 diff --git a/tests/unit/test_protocol_method_files.py b/tests/unit/test_protocol_method_files.py new file mode 100644 index 0000000..4fb971b --- /dev/null +++ b/tests/unit/test_protocol_method_files.py @@ -0,0 +1,193 @@ +"""The catalog serialization, held to the TypeScript twin edge for edge. + +`mthds/protocol/method_files.py` mirrors `mthds-js`'s `protocol/method_files.ts`, and a port of a +format like this one fails quietly: it round-trips its own output and still drops a file the twin +keeps, keeps one the twin drops, or writes bytes the twin would not. So the cases here are not what +the port was expected to do but what the twin was observed to do — `MethodFileCases` records the +bytes and the verdicts a run of the twin printed — and the happy round-trip is the least of them. +""" + +import json +import unicodedata + +import pytest +from pydantic import ValidationError + +from mthds.protocol.exceptions import PipelineRequestError +from mthds.protocol.method_files import _ECMASCRIPT_TRIM_CHARS, MethodFile, parse_method_files, serialize_method_files +from tests.unit.test_data import MethodFileCases + +_TWIN_FILES = [MethodFile.model_validate(entry) for entry in MethodFileCases.TWIN_FILES] + + +class TestMethodFiles: + def test_serialization_writes_the_twins_bytes(self): + # Same value is not the bar; same string is. `json.dumps` defaults — spaced separators, + # `\uXXXX` for anything outside ASCII — decode to the same files and store differently. + assert serialize_method_files(_TWIN_FILES) == MethodFileCases.TWIN_SERIALIZED + + def test_the_serialized_form_is_the_named_array_in_stored_order(self): + assert json.loads(serialize_method_files(_TWIN_FILES)) == MethodFileCases.TWIN_FILES + reversed_files = list(reversed(_TWIN_FILES)) + assert json.loads(serialize_method_files(reversed_files)) == list(reversed(MethodFileCases.TWIN_FILES)) + + def test_no_files_serializes_to_the_empty_string_and_never_to_the_literal_array(self): + # `""` is the platform's "no source" / "clear the field" sentinel. `"[]"` would be stored + # as a source; the platform does guard that literal explicitly and return no contents, so + # the reason for preferring `""` is that it is the sentinel the platform writes, not that + # `"[]"` would reach the TOML parser. + assert serialize_method_files([]) == "" + assert serialize_method_files([MethodFile(name="empty.py", content=" ")]) == "" + + def test_an_array_of_nothing_but_blank_files_collapses_to_the_sentinel(self): + # The round trip that reaches the sentinel, which is the whole reason the module exists: + # a catalog whose every entry is blank parses to no files and re-serializes to `""`. + source = json.dumps([{"name": "a.py", "content": ""}, {"name": "b.py", "content": "  "}]) + assert parse_method_files(source) == [] + assert serialize_method_files(parse_method_files(source)) == "" + + def test_serialization_drops_blank_entries_and_keeps_the_rest(self): + files = [MethodFile(name="a.py", content="x"), MethodFile(name="b.py", content=""), MethodFile(name="c.py", content="y")] + assert json.loads(serialize_method_files(files)) == [{"name": "a.py", "content": "x"}, {"name": "c.py", "content": "y"}] + + @pytest.mark.parametrize(("topic", "content", "is_blank"), MethodFileCases.BLANK_PREDICATE) + def test_blank_is_what_the_twin_calls_blank(self, topic: str, content: str, is_blank: bool): + # The predicate is ECMAScript `trim`'s set, spelled out. `str.strip()` would pass the + # happy cases and fail exactly these: the BOM it keeps, U+001C to U+001F and U+0085 it drops. + # Both directions go through a catalog array, because a bare control character is simply + # not valid JSON — asserting it is refused would prove nothing about the blank predicate. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + catalog = json.dumps([{"name": "f", "content": content}]) + if is_blank: + assert serialized == "", topic + assert parse_method_files(content) == [], topic + assert parse_method_files(catalog) == [], topic + else: + assert json.loads(serialized) == [{"name": "f", "content": content}], topic + assert parse_method_files(catalog) == [MethodFile(name="f", content=content)], topic + + def test_the_trim_set_is_ecmascripts_whole_trim_set(self): + # `BLANK_PREDICATE` samples code points; this derives the set, so a code point dropped + # from the middle of `_ECMASCRIPT_TRIM_CHARS` cannot pass the suite while silently + # reclassifying a file from blank to content. The authority is ECMAScript's own + # grammar — WhiteSpace (TAB, VT, FF, ZWNBSP, and every Space_Separator) plus the four + # LineTerminators — not either implementation's behaviour on a sample. + space_separators = {chr(code_point) for code_point in range(0x110000) if unicodedata.category(chr(code_point)) == "Zs"} + whitespace = space_separators | {chr(0x09), chr(0x0B), chr(0x0C), chr(0xFEFF)} + line_terminators = {chr(0x0A), chr(0x0D), chr(0x2028), chr(0x2029)} + assert set(_ECMASCRIPT_TRIM_CHARS) == whitespace | line_terminators + + @pytest.mark.parametrize(("topic", "content", "expected"), MethodFileCases.LONE_SURROGATES) + def test_an_unpaired_surrogate_is_escaped_as_the_twin_escapes_it(self, topic: str, content: str, expected: str): + # `ensure_ascii=False` writes an unpaired surrogate raw, and the resulting string cannot + # be encoded as UTF-8 at all — so it breaks at the storage boundary, not here, and the + # at-rest bytes differ from the twin's. The twin has escaped it since ES2019. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + assert serialized == expected, topic + assert serialized.encode("utf-8"), topic + assert parse_method_files(serialized) == [MethodFile(name="f", content=content)], topic + + def test_parsing_reads_the_twins_bytes_back_into_files(self): + assert parse_method_files(MethodFileCases.TWIN_SERIALIZED) == _TWIN_FILES + + @pytest.mark.parametrize(("topic", "source"), MethodFileCases.NO_FILES_SOURCES) + def test_a_blank_source_and_an_empty_array_are_both_no_files(self, topic: str, source: str | None): + assert parse_method_files(source) == [], topic + + def test_parsing_drops_blank_entries_so_the_round_trip_is_stable(self): + source = json.dumps([{"name": "a.py", "content": "x"}, {"name": "b.py", "content": " "}]) + assert parse_method_files(source) == [MethodFile(name="a.py", content="x")] + + def test_parsing_tolerates_and_strips_an_extra_member(self): + # The wire may carry more than the two members; the typed surface carries exactly two. + # Validating an entry through the closed model would refuse what the twin accepts. + assert parse_method_files('[{"name":"a.py","content":"x","extra":1}]') == [MethodFile(name="a.py", content="x")] + + def test_parsing_keeps_order_duplicates_and_blank_names(self): + # A blank name is not a reason to drop an entry — the platform's decoder disagrees, and the + # port follows the twin so the three implementations do not split two against one. + source = '[{"name":"z","content":"1"},{"name":"a","content":"2"},{"name":"a","content":"3"},{"name":"","content":"4"}]' + assert parse_method_files(source) == [ + MethodFile(name="z", content="1"), + MethodFile(name="a", content="2"), + MethodFile(name="a", content="3"), + MethodFile(name="", content="4"), + ] + + def test_round_trip_serialize_then_parse(self): + assert parse_method_files(serialize_method_files(_TWIN_FILES)) == _TWIN_FILES + assert parse_method_files(serialize_method_files([])) == [] + + @pytest.mark.parametrize(("topic", "source"), MethodFileCases.CONTRACT_VIOLATIONS) + def test_anything_but_the_named_array_is_a_contract_violation(self, topic: str, source: str): + with pytest.raises(PipelineRequestError) as exc_info: + parse_method_files(source) + assert "method file" in str(exc_info.value).lower(), topic + + def test_a_source_that_exhausts_the_decoder_is_a_contract_violation(self): + # `json.loads` raises `RecursionError` rather than `JSONDecodeError` on a source nested + # deeper than its decoder allows, and `RecursionError` is not a `ValueError`, so catching + # only `JSONDecodeError` let it escape this typed surface. + # + # The depth that trips it is an interpreter build constant, not `sys.getrecursionlimit()`: + # it rose by an order of magnitude in CPython 3.14 and differs even between patch + # releases, so pinning one depth pins an accident of one build. Depth 10000 refuses on + # 3.11 to 3.13 and parses cleanly on 3.14, which is a supported interpreter the CI matrix + # runs — so the depth is discovered here instead. + # + # This guard is best-effort and NOT a parity rule: the twin's `JSON.parse` is iterative + # and accepts every depth tried, so it is the module's one forced divergence from it. + for depth in (10_000, 200_000, 500_000): + source = '[{"name":"a","content":"x","extra":' + "[" * depth + "]" * depth + "}]" + try: + json.loads(source) + except RecursionError: + with pytest.raises(PipelineRequestError): + parse_method_files(source) + return + pytest.skip("no tried nesting depth exhausts this interpreter's JSON decoder") + + @pytest.mark.parametrize(("topic", "content", "expected"), MethodFileCases.SURROGATE_PAIRS) + def test_a_surrogate_pair_is_written_as_the_astral_character_the_twin_writes(self, topic: str, content: str, expected: str): + # Python addresses code points and so can spell an astral character two ways; the twin + # addresses UTF-16 code units and has only one spelling. Escaping both halves instead of + # combining them makes the at-rest bytes differ from the twin's on every one of the pairs, + # and costs serialization its idempotence: the parse combines what the dump split. + serialized = serialize_method_files([MethodFile(name="f", content=content)]) + assert serialized == expected, topic + assert serialized.encode("utf-8"), topic + assert serialize_method_files(parse_method_files(serialized)) == expected, topic + + def test_the_surrogate_rules_cover_the_name_as_well_as_the_content(self): + # One `json.dumps` writes both members, so both get the same treatment — pinned here + # because a fix applied to only one of them would still pass every content-only case. + pair = chr(0xD83D) + chr(0xDE42) + assert serialize_method_files([MethodFile(name=pair, content="x")]) == '[{"name":"\U0001f642","content":"x"}]' + lone = "a" + chr(0xD800) + "b" + assert serialize_method_files([MethodFile(name=lone, content="x")]) == '[{"name":"a\\ud800b","content":"x"}]' + + def test_an_oversized_integer_in_an_ignored_member_is_accepted_as_the_twin_accepts_it(self): + # Python runs every JSON integer through `int()`, which refuses a literal longer than + # `sys.get_int_max_str_digits()` with a bare `ValueError` — neither a `JSONDecodeError` nor + # a `RecursionError` — so it escaped this typed surface entirely, while the twin parsed the + # same bytes and stripped the member. Integers are decoded as floats here, which is the + # twin's own number model and immune to the limit. + source = '[{"name":"a.py","content":"x","extra":' + "1" * MethodFileCases.OVERSIZED_INTEGER_DIGITS + "}]" + # The vector is only meaningful while Python's own decoder still refuses these digits. + with pytest.raises(ValueError, match="Exceeds the limit"): + json.loads(source) + assert parse_method_files(source) == [MethodFile(name="a.py", content="x")] + + def test_an_oversized_integer_where_a_string_belongs_is_still_a_contract_violation(self): + # Accepting the number must not accept the entry: the twin refuses this one too. + source = '[{"name":"a.py","content":' + "1" * MethodFileCases.OVERSIZED_INTEGER_DIGITS + "}]" + with pytest.raises(PipelineRequestError): + parse_method_files(source) + + def test_the_typed_shape_is_closed_and_strict(self): + # Leniency lives in the parser, not the type: a consumer constructing a file gets exactly + # `name` and `content`, both strings. + with pytest.raises(ValidationError): + MethodFile.model_validate({"name": "a.py", "content": "x", "extra": 1}) + with pytest.raises(ValidationError): + MethodFile.model_validate({"name": 1, "content": "x"}) diff --git a/tests/unit/test_protocol_options.py b/tests/unit/test_protocol_options.py new file mode 100644 index 0000000..5b2083a --- /dev/null +++ b/tests/unit/test_protocol_options.py @@ -0,0 +1,212 @@ +"""Unit tests for the run-source argument surface — the predicates every Python client shares.""" + +import pytest + +from mthds.protocol.exceptions import PipelineRequestError +from mthds.protocol.options import ( + RUN_ARG_BUNDLE_B64, + RUN_ARG_FILES, + RUN_ARG_METHOD_ID, + RUN_ARG_METHOD_REF, + assert_exclusive_run_sources, + assert_method_ref_pairs_with_nothing, + has_bundle_payload, + normalized_selector, + run_selector_extensions, +) + +_ADDRESS = "github.com/Pipelex/methods/documents@v0.1.0" + + +class TestRunSourceOptions: + """The run-source predicates and the selector normalization that goes with them.""" + + # ── wire keys ──────────────────────────────────────────────── + + def test_keys_match_the_wire(self) -> None: + """The constants are the request's own property names — the runners read `extra` by them.""" + assert (RUN_ARG_FILES, RUN_ARG_BUNDLE_B64, RUN_ARG_METHOD_REF, RUN_ARG_METHOD_ID) == ( + "files", + "bundle_b64", + "method_ref", + "method_id", + ) + + # ── normalized_selector ────────────────────────────────────── + + def test_absent_selector_is_none(self) -> None: + """An absent selector normalizes to None.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value=None) is None + + def test_empty_selector_is_none(self) -> None: + """An empty selector selects nothing, so it normalizes to None rather than riding the wire.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value="") is None + + def test_present_selector_passes_through(self) -> None: + """A non-empty string is returned unchanged.""" + assert normalized_selector(name=RUN_ARG_METHOD_REF, value=_ADDRESS) == _ADDRESS + + @pytest.mark.parametrize("wrong_value", [123, ["mt_1"], 0, [], {}, True]) + def test_non_string_selector_is_refused(self, wrong_value: object) -> None: + """A non-string is refused rather than dropped (falsy) or forwarded (truthy) — one answer + for every wrong type. + + This DIVERGES from the JS lineage rather than matching it: `nonEmptyString` in + `pipelex-sdk-js` never throws, so it drops most of these silently and forwards + `["mt_1"]`, whose `length` happens to be non-zero. Refusing is the better behaviour; + the two sides do not agree on it yet. + """ + with pytest.raises(PipelineRequestError, match="must be a string, received"): + normalized_selector(name=RUN_ARG_METHOD_REF, value=wrong_value) + + def test_selector_error_names_the_argument_and_the_type(self) -> None: + """The message names the argument the caller passed and the type they passed for it.""" + with pytest.raises(PipelineRequestError, match=r"method_id must be a string, received int\."): + normalized_selector(name=RUN_ARG_METHOD_ID, value=7) + + # ── run_selector_extensions ────────────────────────────────── + + def test_no_selector_contributes_nothing(self) -> None: + """Neither selector present: the contribution is empty, so the caller's `extra` is untouched.""" + assert run_selector_extensions() == {} + + def test_both_selectors_ride_under_their_wire_keys(self) -> None: + """Each selector reaches the mapping under its own wire key.""" + assert run_selector_extensions(method_ref=_ADDRESS, method_id="mt_abc") == { + RUN_ARG_METHOD_REF: _ADDRESS, + RUN_ARG_METHOD_ID: "mt_abc", + } + + def test_empty_selectors_are_dropped(self) -> None: + """An empty selector contributes nothing (presence is decided after normalization).""" + assert run_selector_extensions(method_ref="", method_id="") == {} + + def test_extensions_refuse_a_non_string_selector(self) -> None: + """The normalization is the same one `normalized_selector` applies.""" + with pytest.raises(PipelineRequestError, match="method_id must be a string"): + run_selector_extensions(method_id=42) + + # ── has_bundle_payload ─────────────────────────────────────── + + def test_no_bundle(self) -> None: + """Neither encoding present: nothing to run from a bundle.""" + assert has_bundle_payload() is False + + def test_files_payload(self) -> None: + """A non-empty file map carries a method.""" + assert has_bundle_payload(files={"main.mthds": 'domain = "answer"'}) is True + + def test_zip_payload(self) -> None: + """A non-empty base64 zip carries a method.""" + assert has_bundle_payload(bundle_b64="UEsDBA==") is True + + def test_empty_encodings_carry_no_method(self) -> None: + """Unlike exclusivity, this keys off a RUNNABLE payload: empty encodings carry nothing.""" + assert has_bundle_payload(files={}, bundle_b64="") is False + + # ── assert_exclusive_run_sources ───────────────────────────── + + def test_nothing_present_is_legal(self) -> None: + """Exclusivity says nothing about an absent source — the precondition is a separate check.""" + assert_exclusive_run_sources() + + def test_contents_alone_is_legal(self) -> None: + """Inline contents alone is the protocol's own run source.""" + assert_exclusive_run_sources(mthds_contents=['domain = "answer"']) + + def test_files_alone_is_legal(self) -> None: + """One bundle encoding alone is legal.""" + assert_exclusive_run_sources(files={"main.mthds": 'domain = "answer"'}) + + def test_two_encodings_are_refused(self) -> None: + """`files` and `bundle_b64` are two encodings of one bundle.""" + with pytest.raises(PipelineRequestError, match="two encodings of the same bundle"): + assert_exclusive_run_sources(files={"main.mthds": "x"}, bundle_b64="UEsDBA==") + + def test_two_encodings_key_off_presence_not_emptiness(self) -> None: + """An empty `files` map beside a zip still expressed two encodings.""" + with pytest.raises(PipelineRequestError, match="two encodings of the same bundle"): + assert_exclusive_run_sources(files={}, bundle_b64="UEsDBA==") + + def test_bundle_with_contents_is_refused(self) -> None: + """A bundle carries its own `.mthds`, so it cannot be combined with inline contents.""" + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], files={"main.mthds": "x"}) + + def test_zip_with_contents_is_refused(self) -> None: + """The same rule holds for the zip encoding.""" + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], bundle_b64="UEsDBA==") + + def test_empty_contents_beside_a_bundle_is_legal(self) -> None: + """An empty contents list is "no contents", so it does not collide with a bundle.""" + assert_exclusive_run_sources(mthds_contents=[], files={"main.mthds": "x"}) + + def test_bundle_with_contents_keys_off_presence_not_emptiness(self) -> None: + """An empty `files` map beside inline contents is still two run sources. + + This arm and `has_bundle_payload` deliberately read the same value differently: an empty + encoding is PRESENT for exclusivity and RUNNABLE for neither. Pinning both sides of that + disagreement is what stops a later refactor from routing exclusivity through + `has_bundle_payload` and turning this refusal into an accepted request. + """ + assert has_bundle_payload(files={}) is False + with pytest.raises(PipelineRequestError, match="self-contained"): + assert_exclusive_run_sources(mthds_contents=['domain = "answer"'], files={}) + + # ── assert_method_ref_pairs_with_nothing ───────────────────── + + def test_absent_method_ref_checks_nothing(self) -> None: + """With no address, every other combination is somebody else's rule.""" + assert_method_ref_pairs_with_nothing(mthds_contents=['domain = "answer"'], method_id="mt_abc") + + def test_empty_method_ref_checks_nothing(self) -> None: + """An empty address selects nothing, so it is not an address run.""" + assert_method_ref_pairs_with_nothing(method_ref="", mthds_contents=['domain = "answer"'], method_id="mt_abc") + + def test_method_ref_alone_is_legal(self) -> None: + """The address alone is a complete run source.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS) + + def test_method_ref_with_contents_is_refused(self) -> None: + """An address and inline contents are two run sources.""" + with pytest.raises(PipelineRequestError, match="inline mthds_contents are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, mthds_contents=['domain = "answer"']) + + def test_method_ref_with_files_is_refused(self) -> None: + """An address and a bundle are two run sources — the arm the private SDK copy lacked.""" + with pytest.raises(PipelineRequestError, match=r"a method bundle \(bundle_b64 / files\) are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, files={"main.mthds": "x"}) + + def test_method_ref_with_zip_is_refused(self) -> None: + """The bundle arm keys off presence, so the zip encoding is refused too.""" + with pytest.raises(PipelineRequestError, match=r"a method bundle \(bundle_b64 / files\) are mutually exclusive"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, bundle_b64="UEsDBA==") + + def test_method_ref_with_method_id_is_refused(self) -> None: + """An address run has its own provenance and takes no run-history linkage id.""" + with pytest.raises(PipelineRequestError, match="takes no run-history linkage id"): + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, method_id="mt_abc") + + def test_method_ref_with_empty_method_id_is_legal(self) -> None: + """An empty id selects nothing, so it is not a second selector.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, method_id="") + + def test_empty_contents_beside_method_ref_is_legal(self) -> None: + """An empty contents list is "no contents" — the presence semantics the server uses.""" + assert_method_ref_pairs_with_nothing(method_ref=_ADDRESS, mthds_contents=[]) + + def test_non_string_method_ref_is_refused(self) -> None: + """The address goes through the same boundary normalization as any selector.""" + with pytest.raises(PipelineRequestError, match="method_ref must be a string"): + assert_method_ref_pairs_with_nothing(method_ref=["github.com/Pipelex/methods"]) + + def test_non_string_method_id_is_refused_without_an_address(self) -> None: + """The selector type refusal does not depend on an unrelated argument. + + `method_id` is normalized before the no-address early return, so one wrong value gets one + answer whether or not a `method_ref` accompanies it — the unconditional guarantee + `docs/runners.md` advertises for a selector at this boundary. + """ + with pytest.raises(PipelineRequestError, match=r"method_id must be a string, received int\."): + assert_method_ref_pairs_with_nothing(method_id=42) diff --git a/tests/unit/test_protocol_stuff.py b/tests/unit/test_protocol_stuff.py new file mode 100644 index 0000000..c40bda2 --- /dev/null +++ b/tests/unit/test_protocol_stuff.py @@ -0,0 +1,73 @@ +"""Unit tests for the protocol's stuff and concept shapes — what a stuff says about its concept.""" + +import json + +import pytest +from pydantic import ValidationError + +from mthds.protocol.concept import ConceptAbstract +from mthds.protocol.stuff import StuffAbstract, StuffContentAbstract + + +class _StubConcept(ConceptAbstract): + """A runtime's concept: the protocol's two fields, plus whatever the runtime needs in memory.""" + + structure_class_name: str + + +class _StubContent(StuffContentAbstract): + value: str + + +class _StubStuff(StuffAbstract[_StubConcept, _StubContent]): + pass + + +def _stub_stuff() -> _StubStuff: + return _StubStuff( + stuff_code="stuff_1", + stuff_name="analysis", + concept=_StubConcept(code="ContractAnalysis", domain_code="legal", structure_class_name="ContractAnalysis"), + content=_StubContent(value="high"), + ) + + +class TestStuffConceptSerialization: + def test_concept_abstract_carries_the_reference_only(self) -> None: + """A concept's definition belongs to the library, so the protocol model holds the two fields + that name it and nothing else. + """ + assert set(ConceptAbstract.model_fields) == {"code", "domain_code"} + + def test_definition_fields_are_refused(self) -> None: + """The trimmed fields are not silently tolerated: the model is closed, so a payload + carrying a definition fails the parse rather than smuggling it beside a stuff. + """ + with pytest.raises(ValidationError, match="description"): + ConceptAbstract.model_validate({"code": "ContractAnalysis", "domain_code": "legal", "description": "A contract analysis"}) + + def test_concept_ref_is_the_domain_qualified_reference(self) -> None: + assert _stub_stuff().concept.concept_ref == "legal.ContractAnalysis" + + def test_dump_emits_the_concept_as_its_reference(self) -> None: + """`model_dump()` gives the ref string, not the concept object — including what the + runtime's own subclass carries beside the two protocol fields. + """ + dumped = _stub_stuff().model_dump() + + assert dumped["concept"] == "legal.ContractAnalysis" + assert dumped["content"] == {"value": "high"} + assert dumped["stuff_code"] == "stuff_1" + assert dumped["stuff_name"] == "analysis" + + def test_json_dump_emits_the_concept_as_its_reference(self) -> None: + """JSON mode agrees with python mode — the wire form is the protocol's on both.""" + assert json.loads(_stub_stuff().model_dump_json())["concept"] == "legal.ContractAnalysis" + + def test_serialization_schema_declares_the_concept_a_string(self) -> None: + """A server embedding a stuff in a response publishes `concept` as a string, so the + generated OpenAPI says what the wire actually carries. + """ + schema = _StubStuff.model_json_schema(mode="serialization") + + assert schema["properties"]["concept"] == {"type": "string"} diff --git a/uv.lock b/uv.lock index 32ba765..e9c7857 100644 --- a/uv.lock +++ b/uv.lock @@ -212,7 +212,7 @@ wheels = [ [[package]] name = "mthds" -version = "0.14.0" +version = "0.15.0" source = { editable = "." } dependencies = [ { name = "httpx" },