A saved method holds a method tree: the files a method is made of, each at its tree path, the same tree a run and a validation take as files. The draft holds one, and each published version holds the one it was published from. A tree is named by its tree digest, sha256: and the SHA-256 of its listing, so two methods holding the same files hold the same digest. Both are defined by the MTHDS standard (mthds/docs/spec/method-tree.md), and this SDK reads and writes them through mthds's own library, mthds.method_tree, so the tree a save sends is the tree a run or a validate would send.
The platform keeps each file once per organization, as a blob named by its SHA-256, so a save sends only the files the organization does not hold yet, and a read lists the files with a short-lived link to each one's bytes.
create_method(MethodWriteInput) and write_draft(method_id, MethodDraftInput) take the tree in one of three forms:
| Form | What it is | How it is sent |
|---|---|---|
files |
The standard's map of file entries (mthds.method_tree.file_entry.FileEntry): a text, {"base64": …}, or {"sha256": …, "size": …} for bytes the organization already holds. |
As given. |
tree |
A tree listed from a directory or a .mthds file by list_method_tree, or the path of that directory or file, a Path or a str, which the client lists the same way. |
Inline when small, by hash otherwise (below). |
contents |
Each file's bytes, or its text, by tree path. | Inline when small, by hash otherwise (below). |
A tree given as tree or contents is sent inline when its inline map weighs 256 KiB or less, written as JSON with no whitespace, or when the platform keeps no blob store, which its version handshake says (blob_store). A tree is held to the inline limit the platform announces too, limits.method_tree.max_inline_bytes in that handshake, when it is lower: the client asks the handshake once for each request carrying a tree, and the same answer gives the limits a path is listed under and serves mthds's preparation, which asks it for a tree past the threshold. contents takes no handshake of its own and keeps the 256 KiB threshold alone, so a small one costs no round trip, as @pipelex/sdk sends one. Otherwise the client asks POST /v1/blobs/missing which files the organization lacks, naming the tree's digest, uploads each one it lacks once, straight to the upload target the platform names, and sends every file by hash. This is mthds's prepare_files and prepare_files_from_contents, run before the save. A file uploaded once is never uploaded again, by this method or any other of the organization.
An answer of POST /v1/blobs/missing this SDK cannot read, a body that is not JSON or not the route's answer, is the ApiResponseError of an answer the SDK cannot read, as on every other route. So is an answer asking for a blob the tree does not hold, which is the platform's fault and not the caller's: client.response_unreadable, raised before any upload. Its upload targets are signed links, so response_body carries the answer with every link redacted, and the error chains at most a copy of pydantic's refusal whose values are redacted the same way. A call to prepare_files or prepare_files_from_contents of its own reads these answers the same way.
A text in contents is encoded in UTF-8, and a text a file cannot hold, one carrying a lone surrogate, is refused. A form of the wrong type is refused with RequestArgumentError before any request, naming the argument: a files or a contents that is not a map (not_a_map), a value in contents that is neither bytes nor text (entry_mistyped), and a tree that is neither a MethodTree nor a path (not_a_tree). Every file of a tree listed from a directory is read again before it is sent and checked against the hash its listing recorded, so a file changed since the listing raises mthds's TreeSourceError rather than send another tree than the one listed.
from pathlib import Path
from pipelex_sdk.product_models import MethodDraftInput, MethodWriteInput
tree = await client.list_method_tree(Path("methods/receipts"))
method = await client.create_method(MethodWriteInput(name="Receipts", tree=tree))
draft = await client.write_draft(
method.method_id,
MethodDraftInput(contents={"main.mthds": source, "helpers/fx.py": helper}, expected_updated_at=method.updated_at),
)A draft write replaces the whole tree. write_draft refuses a draft that gives no tree, and an explicit files=None empties the draft. create_method with no tree creates a method with an empty draft. The client refuses two forms at once, and contents holding no file, with a RequestArgumentError before anything is sent; mthds refuses a tree holding no file the same way.
The save is refused, with the ApiResponseError of its problem, when the platform will not take the tree:
Status and code |
When |
|---|---|
422 method.package_invalid |
The map breaks the standard's rules, a path that is not a tree path or two paths that collide among them; the problem names the path. |
422 method.package_invalid |
The tree passes one of the deployment's limits, which the problem's params name (reason too_large and the limit). |
422 request.invalid |
A hash entry states another size than the blob the organization holds under that hash. |
422 request.blobs_missing |
A hash entry names bytes the organization does not hold. A save built from tree or contents uploads every blob it names first, so it meets this only if a blob was removed between the upload and the save; the client does not retry it. |
413 request.too_large |
The method would leave no room for the summary a publish adds. |
An upload that fails raises MethodFileTransferError with operation upload, and the method is not saved (below).
The crate routes, resolve, codegen and pipe_io, and prepare_inputs, which reads pipe_io, take the method's tree in the same three forms and send a tree or contents by the same rules, so a tree too large to send inline is uploaded once and then read by hash. There an upload that fails means the request was not sent (architecture.md, "Crate routes").
execute, start, start_and_wait and validate take the method's tree in the same three forms, files second as in the protocol client and tree and contents by name, and send a tree or contents by the same rules before the run's request, so a run of a large method sends a small request and uploads only the files the organization does not hold yet. A directory is listed with the standard's lister, under the deployment's limits, so a run's inputs.json or inputs.toml, at any depth, is never sent. The tree is given in one form, never beside a method_ref, and a refused call lists, uploads and sends nothing. An upload that fails raises MethodFileTransferError, and no run is started (for validate, the request is not sent). start_and_wait prepares the tree once and hands the same map to the start and, against a runner with no run store, to the blocking execute. The platform refuses a run's tree with the 422s of the table above, and validates a tree only for a caller acting in an organization.
from pathlib import Path
method_dir = Path("methods/receipts")
report = await client.validate(tree=method_dir)
results = await client.start_and_wait(tree=method_dir, inputs={"topic": "quantum computing"})To send one tree to several calls, a validation, a pipe I/O read and a run for instance, prepare it once with the base client's own calls, files = await client.prepare_files(await client.list_method_tree(method_dir)), and pass files=files to each: the map is then sent as given, with no further listing or round trip. A failed upload there raises mthds's BlobUploadError, the call being the standard client's own.
method_tree_limits() returns the limits the deployment holds a tree to, as mthds.method_tree.models.TreeLimits: the most files, the largest file and the largest total, read from limits.method_tree of the public version handshake (GET /v1/version). A limit the server does not announce is left unset, and none is ever assumed, so a bare runner, which announces none, yields no limit. list_method_tree(path) lists a tree under those limits, off the event loop, so a tree the platform would refuse for its size is refused while it is listed, with mthds's TreeLimitError naming the limit, before any file is hashed.
get_method(method_id) answers a MethodDetail: the method, with files, the draft's tree listed in tree path order. get_method_version(method_id, version) answers a MethodVersion: the version's summary, with the same files. Each file is a SavedMethodFile:
| Field | Meaning |
|---|---|
path |
Its tree path. |
sha256 |
The SHA-256 of its bytes, 64 lower-case hex digits. |
size |
Its size in bytes. |
url |
A link to its bytes, signed by storage, which any HTTP client reads with no credential. |
expires_at |
When the link stops working, about fifteen minutes after the read. |
An empty draft lists no file and has a draft_digest of None. The listing pages are lighter: list_methods rows and list_method_versions summaries carry the digest and no files.
A link is a bearer credential. Anyone holding it reads the file until it expires, so it belongs in no log, message or stored record; this SDK puts it in none of its errors, and a SavedMethodFile's repr leaves it out. A read that fails because a link expired succeeds once the method is read again.
An answer of these two routes the SDK cannot read, a listing whose sha256 is not 64 lower-case hex digits or whose size is negative among them, is an ApiResponseError that reaches no link either: response_body has every url member and every link's query replaced by [redacted], and the error's chain holds no failure that holds the body. A pydantic refusal is chained as a copy keeping each error's type and location, so a caller can still read which member was refused, its values redacted the same way; a body that is not JSON chains nothing, since json.JSONDecodeError holds the whole document. Neither the cause nor the context of the error is the original failure, so no traceback prints it.
A method whose draft names a tree the platform holds no record of answers get_method with a 500 runtime.unexpected, and a version stored before methods were trees and never rewritten as one names no tree to serve, so get_method_version answers it with 500 runtime.unexpected too.
write_method_tree(files, destination=…) writes a listing to disk, each file at its tree path under destination:
detail = await client.get_method("mt_123")
await client.write_method_tree(detail.files, destination=Path("receipts"))Each file is downloaded through its link, at most eight at a time, by an HTTP client apart from the API client, so neither the API key nor any header of this client goes to storage. A file holds its turn until its bytes are on disk, so no more than eight are held in memory at once, and each download runs under request_timeout_seconds for its whole exchange, the body included, so a body that stalls once its headers arrived times out too. Each file's size and SHA-256 are checked against the listing, so what lands on disk is the tree the digest names. The first failure stops the write, the downloads still running cancelled. The files are written into a fresh hidden directory beside destination, which takes the name destination only once every file is written, so a failed, interrupted or cancelled write leaves nothing under that name and removes what it wrote. The checks on disk, the directories, the move into place and the cleanup run off the event loop.
destination is made absolute and normalized first, without following a symbolic link, so a relative one, Path(".") included, names the directory it means, and the hidden directory sits beside it rather than inside it. It must not exist, or must be an empty directory: one holding anything is refused, rather than written into or replaced, and so is the root of a filesystem. An empty directory is replaced by the written tree's directory, so a process whose working directory it was has to change into it again. A listing holding a path that is not a tree path, the same path twice, or two paths that collide once case-folded or where one names a directory of the other, is refused too. Each refusal is a RequestArgumentError raised before any download. An empty listing writes an empty directory.
MethodFileTransferError is raised when a file could not be moved between this machine and storage. operation says which way (upload for a save, a run, a validation or a crate request, download for write_method_tree), reason why, sha256 which bytes, path the tree path when one is known, and status storage's HTTP status when it answered. Neither its message nor any member carries the link, nor storage's error body, which can echo the signed request, and it keeps neither a cause nor a context: the mthds error an upload refusal starts as quotes that body, and httpx's error for a request that got no answer, or whose body httpx could not decode (DecodingError), holds the request and its URL.
reason |
When | Its problem |
|---|---|---|
unreachable |
Storage gave no answer: the connection failed, the time limit ran out (TimeoutError in the message), or the body could not be decoded. An upload that got no answer carries no sha256, since mthds does not say which blob it was. |
runtime.unavailable, or the problem mthds stated for an upload that got no answer (client.unreachable, client.timeout) |
storage_error |
Storage timed out or throttled the request (408, 429) or failed (a 5xx other than 501). |
runtime.unavailable |
refused |
Storage refused the request with any other status, an expired link among them. Read the method again for fresh links. | client.response_unreadable naming the status, runtime.unsupported for a 501 |
mismatch |
A download's bytes are not the file the listing names: another size, or another SHA-256. | client.response_unreadable |
An upload to a target that asks for a create-only write and answers 412 is not a failure: the bytes are already stored.
A method used to hold its source as two strings, mthds (a bundle, or the JSON text of a file array) and python (the JSON text of the custom PipeFunc files), and a version carried source_digest. Both fields are gone from every read and every write, with no alias kept: a method is a tree, a version's digest is tree_digest, and the publish state is derived from the two digests, the draft unchanged when draft_digest is not None and equals latest_published.tree_digest. method_source_to_contents, MethodFile, parse_method_files and serialize_method_files went with them.
A run's source_digest is something else, and stays: the SHA-256 of the sources a run sent as text, 64 hex digits with no prefix. RunDetail also carries the run's tree_digest and crate_fingerprint, each None for a run that recorded none.