diff --git a/docs/architecture.md b/docs/architecture.md index e01f8e258..b0e9e3462 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -389,9 +389,10 @@ Passes operate on the IR only; they know nothing about source formats or target ### 2.3 Emitters (IR → artifacts) -Emitters are plugins (SDK-per-language, docs, mock servers…). Out of scope for this session -except for the boundary they impose on the IR; the internal shape mirrors what Kiota and oagen -converged on independently: +Emitters are plugins (SDK-per-language, docs, mock servers, and a command-line client planned as an +extension point, `docs/emitters/docs-and-mock.md`…). Out of scope for this session except for the +boundary they impose on the IR; the internal shape mirrors what Kiota and oagen converged on +independently: 1. **Plan** — compute language-*neutral* per-operation and per-type decisions once (is-paginated, error taxonomy, primary-content/response selection, return-shape classification (a success body @@ -415,8 +416,9 @@ though both are post-milestone scope: - **Write/integrate** — regenerating into a live repo alongside hand-written code requires: deterministic file paths, a generation manifest (spec/emitter/config hashes, sorted file - list, and an entity → generated-symbol map **keyed by IR stable IDs**, which survive path - and name churn), file-header provenance that gates pruning (never delete a file lacking the + list, and an entity → generated-symbol map **keyed by IR stable IDs**, which survive a rename + of what they are not built from, a correlation key carrying the rest (#679)), file-header + provenance that gates pruning (never delete a file lacking the generated-by header), ignore-region markers for hand-written islands, and additive-only merging. Additive-only writers also need a staleness check — (previous-revision entities − current entities) ∩ files-on-disk — to surface dead code that pruning cannot touch. @@ -442,10 +444,12 @@ logical exception-kind and fault, doc-URL template), **telemetry** (request- ID and client-telemetry headers), **logging** (a closed lifecycle-event list), **user-agent** construction (identifier template, app-info enrichment), **idempotency injection** (header name, auto-generate rules), **pagination pacing** (auto-page delay), and **request guards** (option -keys that must not appear as params — misuse detection). Delivery model: canonical defaults + -deep-partial per-emitters/per-project overrides. Precedence rule: *declared IR facts win over -policy defaults* — `ErrorCase.Retryable/Fault` and `Operation.Idempotency` come from the spec; -policy fills in where the spec is silent, never the reverse. +keys that must not appear as params — misuse detection), widened by connection-error retries, +`Retry-After` and **redirects**. Delivery model: canonical defaults + deep-partial layers, the +policy a document embeds below the project's and the project's below each emitter's, each with +per-operation overrides (`docs/emitters/contract/runtime-policy.md`). Precedence rule: *declared +IR facts win over policy defaults* — `ErrorCase.Retryable/Fault` and `Operation.Idempotency` come +from the spec; policy fills in where the spec is silent, never the reverse. ### 2.5 Changing the IR diff --git a/docs/emitters/README.md b/docs/emitters/README.md index af00e3d61..70a73acc8 100644 --- a/docs/emitters/README.md +++ b/docs/emitters/README.md @@ -42,7 +42,8 @@ its own, and answers every construct of the [lowering table](refine/lowering-tab - [Target options](contract/options.md): strict, versioned, each default belonging to a version. - [The type table](contract/type-table.md): which target type each IR leaf type becomes. - [Runtime policy](contract/runtime-policy.md) and [shaping hints](contract/shaping-hints.md): the - two inputs that are not the IR. + two inputs that are not the IR, and [the policy a document embeds](contract/embedded-policy.md), + read into each as its lowest layer. **The plan: language-neutral decisions, computed once** @@ -96,7 +97,7 @@ its own, and answers every construct of the [lowering table](refine/lowering-tab **Other artifacts, and how the work is held** - [Docs and mock-server emitters](docs-and-mock.md), [write / integrate](write-integrate.md), - [surface verification](surface-verification.md). + [verifying a generation](verification.md), [surface verification](surface-verification.md). - [Package layout](packages.md), [mandatory tests](testing/mandatory-tests.md), the [behavior server](testing/behavior-server.md), the [milestone plan](milestones.md) and the [open questions](open-questions.md). @@ -234,11 +235,11 @@ The Kotlin targets read the same plan and lower the same facts as Go. |---|---| | 1 IR is the ABI | `EmitterInput.Doc` is the only structural input; `emitters/*` imports `ir`, `ir/irfacts` + shared contract, never `compiler`/`engine`/a sibling target ([the contract](contract/emitter.md), [package layout](packages.md)) | | 2 Lossless, lowered late | all lowering is in `refine` steps building a fresh target AST; the IR is never mutated; unions/enums/discriminators/visibility/pagination/streaming/LRO arrive un-lowered and the step list is the acceptance test ([refine](refine/pipeline.md), [operations](operations/returns.md)) | -| 3 Stable IDs, names presentation | plan, AST `Origin`, manifest, surface, hints, and the collision table are all keyed by IR stable ID; renames are lookups; collisions never touch identity; a name with no words is named from its own characters alone, and placement reads no count, so neither moves when something else changes ([the plan](plan/overview.md), [naming](refine/naming.md), [output units](refine/output-units.md), [write / integrate](write-integrate.md)) | +| 3 Stable IDs, names presentation | plan, AST `Origin`, manifest, surface, hints, and the collision table are all keyed by IR stable ID; a rename that keeps an entity's pointer is a lookup, and one that moves it the correlation key's; collisions never touch identity; a name with no words is named from its own characters alone, and placement reads no count, so neither moves when something else changes ([the plan](plan/overview.md), [naming](refine/naming.md), [output units](refine/output-units.md), [write / integrate](write-integrate.md)) | | 4 Names neutral, emitter cases | `Ident` starts as the `Naming.Canonical` word sequence, the source spelling beside each word for its letter case only; `RenderCasing`/`EscapeReservedWords` are the only casing steps; struct tags come from `WireName`/`WireID`, a separate channel ([naming](refine/naming.md), [emit](emit/printing.md)) | | 5 Pure, typed diagnostics | `Generate` is pure/reentrant/no-stderr/no-I/O; every un-lowerable construct → coded `ir.Diagnostic`; the engine decides fatality ([the contract](contract/emitter.md), [contract rules](contract/emitter.md#contract-rules-the-type-system-and-tests-enforce)) | | 6 Heuristics are policy | `plan.Policy`, each target's versioned options (its type table and naming tables among them), `verify.Severity`, method-name derivation, and shaping hints are injectable and disable-able; the type table's width-picking rows, the acronyms, the character names and the leading words are defaults a configuration replaces; inferred outputs marked `Inferred` ([target options](contract/options.md), [type table](contract/type-table.md), [plan policy](plan/policy.md), [naming](refine/naming.md), [shaping hints](contract/shaping-hints.md), [surface verification](surface-verification.md)) | | 7 Serializable/deterministic | `Plan` + manifest are JSON-serializable, and `Facts` is rebuilt from the document; maps walked sorted, slices source order; identical input ⇒ identical bytes ([who computes the plan](contract/emitter.md#who-computes-the-plan), [the plan](plan/overview.md), [determinism](emit/printing.md#determinism-and-the-velocity-answer), [mandatory tests](testing/mandatory-tests.md)) | | 8 Optionality ≠ nullability | `LowerOptionality` renders all four states plus protobuf `Presence` as a third axis; collapses are diagnosed ([optionality](refine/models.md#optionality--nullability--presence-inv8)) | | 9 Capability surface designed against every format | every `ir.TypeKind` and every operation facet (pagination/streaming/LRO/one-way/messaging) has a lowering path or diagnosed degrade ([step list](refine/steps.md), [lowering table](refine/lowering-table.md)); docs/mock/validation are registry entries over the same `Doc + Plan`; a new language forces no IR or plan change ([docs and mocks](docs-and-mock.md), [M5](milestones.md)) | -| 10 Runtime policy separate | `policy.Policy` tree in `EmitterInput.Policy`, never on `ir.Document`; it parameterises only boilerplate; declared IR facts win over its defaults ([runtime policy](contract/runtime-policy.md)) | +| 10 Runtime policy separate | `policy.Policy` trees, resolved per operation, in `EmitterInput.Policy`, never on `ir.Document`; it parameterises only boilerplate; declared IR facts win over its defaults ([runtime policy](contract/runtime-policy.md)) | diff --git a/docs/emitters/contract/embedded-policy.md b/docs/emitters/contract/embedded-policy.md new file mode 100644 index 000000000..64eb21752 --- /dev/null +++ b/docs/emitters/contract/embedded-policy.md @@ -0,0 +1,80 @@ +# Policy and hints a document embeds + +A document often carries its SDK's configuration as vendor extensions: retries and timeouts at the +document and on an operation, the statuses that count as failures, method and parameter names, the +sub-client an operation belongs to, an operation exposed under a second name, a parameter every +call sends. The compiler keeps each verbatim, an `Unmodeled` entry of reason `vendor_extension` +([unmodeled](../../ir/unmodeled.md)), and left there every emitter would parse them its own way. So +one reader, `emitters/embed`, turns them into the two inputs they configure, [runtime +policy](runtime-policy.md) and [shaping hints](shaping-hints.md), each a layer below the project's. + +```go +package embed + +type Mapping struct { + Keys map[string]Target // an extension's key as Unmodeled holds it, such as + // "openapi:x-sdk-retries", → what it configures + Disabled bool // reads nothing +} + +type Target struct { + Tree string // "policy" | "hint" + Path []string // the member path inside that tree's JSON form, + // such as ["retry", "maxAttempts"] or ["rename"] +} + +type Layer struct { + Policy policy.Layer // the lowest layer of runtime policy + Hints map[emitters.EntityID]emitters.ShapeHint // the lowest layer of hints +} + +func Read(doc *ir.Document, m Mapping) (Layer, []ir.Diagnostic) +``` + +The engine calls `Read` once per document, before it resolves the layers, so an emitter sees only +the resolved policy and hints and never an extension (INV1). + +## What a key configures + +A key configures the node holding it. A `policy` key on the document configures every operation, +and one on an operation configures that operation, `Layer.Policy.Ops` keyed by its `OpID` +([layers](runtime-policy.md#layers-and-operations)). A `hint` key configures the node's own ID, so +a key on a parameter renames that parameter, on an operation that method, and on a service that +client. A key a mapping names on any other node, such as a `policy` key on a property, configures +nothing and is reported. + +The value is decoded into the field `Path` names, as the tree's own JSON form reads it, refusing an +unknown member. A value of the wrong shape is reported as a warning at the extension and never +coerced, since the document means something by the key that the field does not hold. The entry +stays in `Unmodeled` either way, as a promotion leaves it +([promotion](../../ir/unmodeled.md#promotion)), so a consumer that disagrees with the mapping reads +the entry instead. + +## The mapping is a configuration + +The reader runs by default, and its default mapping is empty. The compiler's promotion policy ships +keys whose reading is a convention independent of any generator, such as `x-deprecated-reason` +([`extension-promotion`](../../compilers/openapi.md#extension-promotion)). No key for a retry, a +timeout or an SDK method name has one: each is a generator's own spelling, and its value's shape is +that generator's. So a configuration names the keys its documents use and what each one sets, and a +key it does not name stays where it was. + +A mapping reads a key as `Unmodeled` holds it, the format's prefix included, and only entries of +reason `vendor_extension`, whose key is the source's own spelling. That is the stability #429 asks +the IR to state for such keys. A key under any other reason, such as `no_ir_home`, names a gap the +IR may close, and a mapping naming one is refused. + +## Precedence + +The embedded layer sits below the project's and the emitter's, for policy +([layers](runtime-policy.md#layers-and-operations)) and for hints alike: what a document's author +embeds is a default the SDK's author can override, and the override is reported as an info +diagnostic, so the document's author can see it was overruled. A declared IR fact still beats every +layer ([precedence](runtime-policy.md#precedence--declared-ir-facts-win-hard-rule)). + +## Measured on the golden spec + +The golden spec (spec @319dc756, morphic @b206dc99) carries hints nearly everywhere: a method name +and a group name on 3,761 of its 3,763 operations, a name on 118 parameters and on 30 properties, +each under a generator's own key. It carries no retry or timeout key. A mapping reading those keys +gives each method the name its document chose, which an emitter would otherwise derive. diff --git a/docs/emitters/contract/emitter.md b/docs/emitters/contract/emitter.md index 8dccc2035..aae9f9ec1 100644 --- a/docs/emitters/contract/emitter.md +++ b/docs/emitters/contract/emitter.md @@ -27,11 +27,19 @@ type EmitterInput struct { Doc *ir.Document // the ABI — the ONLY structural input (INV1) Facts *irfacts.Index // what Doc implies: subtypes, usage, recursion (docs/ir/derived-facts.md), indexed ONCE by the engine, shared with the plan (below) Plan *plan.Plan // language-neutral decisions, computed ONCE by the engine (../plan/overview.md) - Policy policy.Policy // runtime/SDK behavior — a SEPARATE tree, never in the IR (INV10; runtime-policy.md) - Hints map[EntityID]ShapeHint// SDK-shaping hints keyed by IR stable ID (shaping-hints.md) + Policy policy.Resolved // runtime/SDK behavior per operation, its layers resolved — + // a SEPARATE tree, never in the IR (INV10; runtime-policy.md, + // embedded-policy.md) + Hints map[EntityID]ShapeHint// SDK-shaping hints keyed by IR stable ID, their layers resolved + // (shaping-hints.md) Scope *Scope // nil = whole document; else "emit only these" — full Doc still // consumed so global decisions stay byte-identical Prior *PriorState // prior manifest + prior surface for write/integrate; nil first run + Samples map[TargetKey]map[ir.OpID]Sample // each SDK target's usage samples, for a docs target + // (../docs-and-mock.md#samples-come-from-the-sdk-emitters) + Routes map[string]RouteTable // every version snapshot's routes, keyed by version, for a + // target whose Capabilities.CrossVersion is set + // (../docs-and-mock.md#routes-and-links) Options TargetOptions // the target's own typed options, decoded and validated before // Generate runs, its type table and naming tables among them // (options.md, type-table.md, ../refine/naming.md) @@ -42,8 +50,24 @@ type EmitterOutput struct { Diagnostics []ir.Diagnostic // typed, coded, provenance-bearing; engine decides fatality Manifest *manifest.Manifest // entity→symbol map keyed by IR stable IDs (../write-integrate.md) Surface *verify.Surface // neutral API-surface projection of the generated output (../surface-verification.md) + Samples map[ir.OpID]Sample // an SDK target's usage sample per operation, for the docs } +// Span is the lines one top-level declaration of a file holds, 1-based and inclusive. +type Span struct { + From, To int + Origin EntityID +} + +// Sample is one operation's call as an SDK target spells it. +type Sample struct { + Language string // the fence's language, such as "go" or "kotlin" + Code string // the call, its values the plan's examples +} + +// RouteTable is one snapshot's routes: each entity's page path and anchor. +type RouteTable map[EntityID]string + // EntityID is the string form of any IR stable ID (TypeID, OpID, PropID, ServiceID, …). // Every IR ID is a `type X string`, so one keyspace addresses them all. type EntityID = string @@ -57,8 +81,11 @@ type Artifact struct { Header HeaderPlacement // where the generated-by provenance header goes (top/after-shebang/none) SkipIfExists bool // scaffolding written once (config, .gitignore) — never overwritten Overwrite bool // regenerate but preserve ignore-regions (default is additive) - Integrate *IntegrateHint // path-prefix stripping / tree-shake root for live-repo merge (../write-integrate.md) + Integrate *IntegrateHint // path-prefix stripping / tree-shake root for live-repo merge, and + // whether the file is merged as blocks (../write-integrate.md) Origins []EntityID // IR IDs whose emission produced this file (drives ID-keyed prune) + Spans []Span // the lines each top-level declaration holds, by its Origin, so a + // toolchain's message maps to an entity (../verification.md) } ``` @@ -97,10 +124,12 @@ the document, which determines it. ## Contract rules the type system and tests enforce -- **Purity / determinism.** No I/O, clock, randomness, or stderr inside `Generate`. Same - `(Doc, Plan, Policy, Hints, Scope, Options)`, under a context that stays live, ⇒ byte-identical - `Artifacts`; `Facts` is a function of `Doc` and adds nothing to the tuple. Maps are walked in - sorted-key order, slices in source order ([T-6](../testing/mandatory-tests.md)). +- **Purity / determinism.** No I/O, clock, randomness, or stderr inside `Generate`. Same `(Doc, + Plan, Policy, Hints, Scope, Prior, Options)`, under a context that stays live, ⇒ byte-identical + `Artifacts`, the prior manifest included since the names a generation shipped are kept + ([naming](../refine/naming.md#collisions-and-the-names-a-generation-shipped)); `Facts` is a + function of `Doc` and adds nothing to the tuple. Maps are walked in sorted-key order, slices in + source order ([T-6](../testing/mandatory-tests.md)). - **Diagnostics are the only input-failure channel.** A construct the emitter cannot lower emits `ir.Diagnostic{Severity, Code:"emitters//…", Message, Provenance}` — never a silent fallback, never a panic ([T-7](../testing/mandatory-tests.md)). Codes live in a stable diff --git a/docs/emitters/contract/registry.md b/docs/emitters/contract/registry.md index fe0069ca4..c4d021bf4 100644 --- a/docs/emitters/contract/registry.md +++ b/docs/emitters/contract/registry.md @@ -39,6 +39,10 @@ type Capabilities struct { LowersKinds []ir.TypeKind // the kinds this emitter has an emit path for WriteIntegrate bool // supports additive merge into a live repo (../write-integrate.md) SurfaceVerify bool // produces a verify.Surface (../surface-verification.md) + CrossVersion bool // reads every version snapshot's routes (../docs-and-mock.md) + Rows []string // the capability-matrix rows it honors, by key + // (../verification.md) + Verify Verify // how its output is built and checked (../verification.md) Protocols []string // "http", "grpc", "message", "graphql", "otp" Lifecycle Lifecycle // "hidden" | "experimental" | "stable" | "sunset"; required Successor TargetKey // the target replacing a sunset one; "" when none does diff --git a/docs/emitters/contract/runtime-policy.md b/docs/emitters/contract/runtime-policy.md index 75e3a9b6e..07a8a7dcf 100644 --- a/docs/emitters/contract/runtime-policy.md +++ b/docs/emitters/contract/runtime-policy.md @@ -7,13 +7,15 @@ SDKs, docs, and mock servers without dragging SDK opinions along. ## The canonical vocabulary -From oagen's production `SdkBehavior`, the best real-world enumeration (architecture.md §2.4): +From oagen's production `SdkBehavior`, the best real-world enumeration (architecture.md §2.4), +widened by the behaviors every SDK has to choose and a caller has to be able to set: ```go package policy type Policy struct { - Retry Retry // retryable status codes, max attempts, Backoff{initial, multiplier, max, jitter} + Retry Retry // when and how a request is sent again (below) + Redirects Redirects // how a redirect is followed (below) Timeout Timeout // default seconds + env override Errors ErrorTax // status→logical exception-kind and fault, doc-URL template, the bound // on a buffered body (../operations/errors.md, ../operations/returns.md) @@ -25,24 +27,103 @@ type Policy struct { Guards RequestGuard // option keys that must NOT appear as params (misuse detection) } +type Retry struct { + Statuses []int // statuses retried where the IR states nothing; default 408, 429, + // 500, 502, 503, 504 + MaxAttempts int // default 3, the first attempt included + Backoff Backoff // {Initial, Multiplier, Max, Jitter} + Throttled Backoff // the backoff after a failure marked throttling + // (../operations/errors.md) + Connection string // a request that got no response: "never" | "idempotent" (default) | + // "always" + RetryAfter bool // default on: wait what a Retry-After header asks, up to Backoff.Max +} + +type Redirects struct { + Max int // default 10; 0 follows none, returning the 3xx as a response +} + func Defaults() Policy func Merge(base Policy, over DeepPartial) Policy // scalars override; ARRAYS REPLACE (the oagen rule) ``` +- **A request that got no response** may have reached the server and acted, so by default it is + sent again only where the operation is idempotent, as `OpPlan.Effects` says + ([OpPlan](../plan/operations.md)). A status answers that question, so `Statuses` apply whatever + the operation does. An idempotency key the runtime generates (`Idempotency`) makes an operation + idempotent only where the IR declares its token, since only then is the server known to honor + one. +- **`Retry-After`** is the server saying when to come back. Honoring it is the default, and it never + waits past `Backoff.Max`, so a server cannot park a caller indefinitely; a value past it ends the + retries and returns the response. +- **A redirect** is followed as RFC 9110 §15.4 says: 307 and 308 send the method and body again, + 301, 302 and 303 send a `GET`. A redirect to another origin never carries the client's + credentials, which no setting changes, since a server cannot vouch for one it is not. + +What a response may hold that the document does not list is not runtime policy, because it changes a +type or a decoder rather than the boilerplate around them, and docs and mocks have to agree with it. +A union a client reads keeps an unknown variant and an enum an unknown value under `plan.Policy`, +and a `oneOf` two variants admit decodes as `plan.Policy` says ([plan policy](../plan/policy.md)). +Unknown members follow each model's `Additional`, a declared fact, so no setting widens a model the +spec closes or closes one it leaves open. + ## Delivery -Canonical `Defaults()` + deep-partial per-emitters/per-project overrides. Arrays replace entirely -(not concat) — an explicit, documented rule. Policy is delivered in `EmitterInput.Policy`; refiners -read it via `Ctx.Policy`, and it shapes only the generated *runtime binding* (the retry loop, -transport config, user-agent string, telemetry headers, pagination pacing) behind the +Canonical `Defaults()` + deep-partial overrides. Arrays replace entirely (not concat) — an +explicit, documented rule. Policy is delivered resolved, per operation, in `EmitterInput.Policy`; +refiners read it via `Ctx.Policy`, and it shapes only the generated *runtime binding* (the retry +loop, transport config, user-agent string, telemetry headers, pagination pacing) behind the `Transport`/`Serializer` abstraction. It never enters the typed structural model ([runtime abstraction](../emit/templates.md#serialization-behind-a-runtime-abstraction)). +## Layers and operations + +An API's operations rarely want one setting each: an export wants a longer timeout, a `POST` no +retry, a rate-limited endpoint its own backoff. So every source of policy is a layer holding a +document-wide override and overrides keyed by operation, merged by the same `Merge` rules: + +```go +type Layer struct { + Document DeepPartial // what the layer sets for every operation + Ops map[ir.OpID]DeepPartial // what it sets for one operation, over its Document +} + +type Resolved struct { + Document Policy // what an operation no layer names is given + Ops map[ir.OpID]Policy // each operation some layer names, resolved whole +} + +func Resolve(layers ...Layer) (Resolved, []ir.Diagnostic) // lowest first: Defaults() is implied +``` + +The layers, lowest first, are the policy a document embeds ([embedded +policy](embedded-policy.md)), the project's, and the emitter's. A setting takes the value of the +highest layer that sets it, for the operation or for the whole document, and within that layer the +operation's value over the document's. So a project that turns retries off turns them off +everywhere, whatever a document embeds for one operation, and a document's own per-operation +setting still beats its own document-wide one. Ordering by scope first would let a document's +per-operation settings outvote a project-wide choice, which then could not be made without +naming every operation. + +A key naming no operation of the document is reported, never ignored, since an operation whose ID +changed would otherwise lose its settings in silence ([a key that matches +nothing](../write-integrate.md#a-key-that-matches-nothing)). Where a higher layer replaces what the +document embeds, an info diagnostic names the setting, both values and the operation, so the +document's author can see it was overruled. + ## Precedence — declared IR facts win (hard rule) `ErrorCase.Retryable/Throttling/Fault` and `Operation.Idempotency` come from the spec; policy fills -in only where the spec is silent — never the reverse. The refiner enforces this by reading the IR -fact first and consulting policy only on `nil`, or for a `Fault`, on `""`. A `policy-precedence` -test asserts a spec `Retryable:false` beats a policy that lists that status as retryable -([T-9](../testing/mandatory-tests.md)). When policy is overridden by a spec fact, an -`Inferred`-marked diagnostic is emitted so nothing is silent. +in only where the spec is silent — never the reverse, and at every layer. The refiner enforces +this by reading the IR fact first and consulting policy only on `nil`, or for a `Fault`, on `""`. A +`policy-precedence` test asserts a spec `Retryable:false` beats a policy that lists that status as +retryable, at a document-wide setting and at an operation's ([T-9](../testing/mandatory-tests.md)). +When policy is overridden by a spec fact, an `Inferred`-marked diagnostic is emitted so nothing is +silent. + +## A setting that cannot apply + +A setting the plan or a target cannot honor is reported, never dropped and never quietly replaced +by a stricter behavior: a `Redirects.Max` a target's transport cannot bound, or a per-operation +`Timeout` on a runtime that keeps one per client. The report is a coded diagnostic of the target's, +naming the setting and the operation, and the engine decides whether it is fatal (INV5). diff --git a/docs/emitters/contract/shaping-hints.md b/docs/emitters/contract/shaping-hints.md index 0592e29c1..778e2f1d7 100644 --- a/docs/emitters/contract/shaping-hints.md +++ b/docs/emitters/contract/shaping-hints.md @@ -2,18 +2,30 @@ oagen's `OperationHint` / rename / remount / split-union-body / url-builder machinery is *SDK ergonomics, not API semantics* — the same category as runtime policy. It lives in -`EmitterInput.Hints`, keyed by **IR stable ID** (never `"METHOD /path"`), so a path rename never -drops a hint (INV3): +`EmitterInput.Hints`, keyed by **IR stable ID** (never `"METHOD /path"`), so a rename of anything +the ID is not built from, such as an `operationId`, never drops a hint (INV3). A path rename gives +the operation a new ID, and its hint is then [a key that matches +nothing](../write-integrate.md#a-key-that-matches-nothing), reported with the entity its correlation +key now names: ```go type ShapeHint struct { Rename string // presentation rename for the entity's symbol; "" = none Remount *ir.GroupID // move an operation under a different sub-client; nil = keep + AlsoMount []ir.GroupID // expose an operation under these sub-clients too, each + // method a second name for the one its own group places SplitUnionBody bool // flatten a request-body union into typed wrapper methods — OPT-IN, // the one sanctioned union→arguments collapse (../refine/unions.md); default off URLBuilder bool // expose a URL-builder variant of the operation + ClientParams []ParamKey // on a service: the parameters its client holds as settings + // (../operations/client.md#client-settings) Extra map[string]ir.RawValue // free-form, forward-compatible per-target ergonomics } + +type ParamKey struct { // a parameter as every operation names it, never by ID alone + In string // "path" | "query" | "header" | "cookie" + Name string // its wire name +} ``` `Extra` is a bare raw-JSON map rather than `ir.RawConfig`: `RawConfig` names *declared protocol @@ -23,3 +35,15 @@ it is emitter input authored alongside the SDK, and the IR never sees it. Method-name *derivation* is itself an injectable policy marked `Inferred`, disable-able and overridable. Per-target naming/compat overlays are the same shape: an emitter input keyed by IR ID, so one IR document drives different compat baselines per language. + +Hints come in layers, as runtime policy does ([layers](runtime-policy.md#layers-and-operations)): +the hints a document embeds ([embedded policy](embedded-policy.md)), then the project's, then the +emitter's, each field taking the highest layer's value. An ID no entity of the document has is +reported, never ignored ([a key that matches +nothing](../write-integrate.md#a-key-that-matches-nothing)). + +`AlsoMount` answers an operation a document exposes under two names, which one `Remount` cannot. +Each sub-client it names gets a method calling the operation, keyed in the manifest under the +operation's ID with the group beside it, as a declaration derived from an entity is ([write / +integrate](../write-integrate.md)), so the surface differ sees one operation reached two ways. + diff --git a/docs/emitters/contract/type-table.md b/docs/emitters/contract/type-table.md index 4ddee05f8..bc27bef58 100644 --- a/docs/emitters/contract/type-table.md +++ b/docs/emitters/contract/type-table.md @@ -71,8 +71,10 @@ type Dependency struct { that type's literals, checks its constraints and knows whether it has methods. Any other type is known only through its row, and how it codes is the target's ([Go](../targets/go/types.md), [Kotlin](../targets/kotlin/types.md)). A literal of a mapped type (a default, a constant, an enum - value) is built by decoding the value's wire form through its codec, and a constraint on it is not - checked by the SDK, which the server still checks, reported once per key as an info + value) is built by decoding the value's wire form through its codec, an enum declared over the + wire form where the target's constants cannot hold the type + ([enums](../refine/enums.md#a-member-on-the-wire)), and a constraint on it is not checked by the + SDK, which the server still checks, reported once per key as an info `emitters//unchecked-constraint`. A row's codec goes with its type, so a declaration standing over the type keeps it by naming that type again rather than defining a new one ([scalars](../refine/scalars.md)), and a type only a user's row names is taken to have methods. diff --git a/docs/emitters/docs-and-mock.md b/docs/emitters/docs-and-mock.md index 96ac56ce7..86ba7661c 100644 --- a/docs/emitters/docs-and-mock.md +++ b/docs/emitters/docs-and-mock.md @@ -3,26 +3,31 @@ Both are `Emitter` implementations consuming the **same `Doc + Plan`** — the payoff of computing the plan once: docs and mocks agree with the SDK by construction, not by re-derivation. -- **Docs emitter (`emitters/docs`).** Reuses the shared plan verbatim (primary response/content, - pagination, return-shape, error taxonomy — exactly what docs must describe) and most of the refine +- **Docs emitter (`emitters/docs`).** Reuses the shared plan verbatim (pagination, return shape, + error taxonomy — what docs must describe; [what the docs render](#what-the-docs-render) says + which responses) and most of the refine step library, but lowers into a `docast` (pages, operation blocks, schema tables, examples) and runs a **reduced pipeline**: no `EscapeReservedWords`, and casing is a presentation choice (SDK-style identifiers or spec-faithful source names, set in its options). The spans [`Docs.Refs`](../ir/annotations.md#references-in-prose) marks resolve to intra-doc links via the *same* IR-ID lookup the Go emitter uses for doc comments, so a link follows what its target renders as. Examples use the schema arm (`Value/Headers`) vs the operation arm - (`Input/Output/Error`) correctly. A union `TypePlan.Open` marks is documented as one a response - may extend, and every `PlannedError` and `PlannedResponse` as the body and headers it declares. No - runtime `Policy` is consulted — proving the IR/policy split pays off. + (`Input/Output/Error`) correctly. A union or enum `TypePlan.Open` marks is documented as one a + response may extend, one `TypePlan.Inclusive` marks as one whose value may be several variants at + once, and every `PlannedError` and `PlannedResponse` as the body and headers it declares. An + enveloped response is documented as the `Type` a method returns and the `Envelope.Body` the wire + carries, and an operation as safe or idempotent where its `Effects` say so. No runtime `Policy` is + consulted — proving the IR/policy split pays off. - **Mock-server emitter (`emitters/mock`).** Reuses `PrimaryResponse`, the content `Return` elects - and the IR `Examples` to synthesize canned responses, `Others` to answer as each declared response - does, the error taxonomy to simulate declared failures, `TypePlan.Open` to say whether a variant - the document does not list may be sent, the `BindingView` to route requests, and the visibility - projections to honor the same wire shapes an SDK sends/receives. Refine lowers into a - server-routing AST (routes from `HTTPBinding.URITemplate`, handlers validating against - constraints). Because the same plan feeds SDK and mock, the two are **wire-consistent by - construction** — the mock can *be* the interception target the SDK's wire-conformance harness - ([mandatory tests](testing/mandatory-tests.md)) is diffed against. + and the IR `Examples` to synthesize canned responses, sending the whole `Envelope.Body` where a + response is enveloped, `Others` to answer as each declared response does, the error taxonomy to + simulate declared failures, `TypePlan.Open` to say whether a variant or an enum value the document + does not list may be sent, the `BindingView` to route requests, and the visibility projections to + honor the same wire shapes an SDK sends/receives. Refine lowers into a server-routing AST (routes + from `HTTPBinding.URITemplate`, handlers validating against constraints). Because the same plan + feeds SDK and mock, the two are **wire-consistent by construction** — the mock can *be* the + interception target the SDK's wire-conformance harness ([mandatory + tests](testing/mandatory-tests.md)) is diffed against. - **Validation emitter (future).** Additionally consumes the validation-only constructs kept verbatim in `Unmodeled` (`not`/`if-then-else`/`dependentSchemas`, …) that SDK/docs emitters ignore. It selects them by `UnmodeledEntry.Reason == ir.ReasonValidationOnly` rather than by key @@ -30,5 +35,138 @@ plan once: docs and mocks agree with the SDK by construction, not by re-derivati `UnmodeledEntry.Provenance` — the keyword's own position, not the enclosing schema's. The contract already carries all of that; no IR change is needed (INV9). +- **CLI emitter (future).** A command-line client is the third surface APIs ship beside SDKs and + docs, and almost all of it is IR and plan facts ([a command-line target](#a-command-line-target)). + Adding any of these is a registry entry over one `Doc + Plan` — no change to the ABI, no change to the IR. + +## What the docs render + +The primary response and content are an SDK's ergonomic choice: a method returns one. A reader of +the docs needs every answer the API gives, so the docs emitter renders every `Response`, every +`ErrorCase` and every `Content` the operation declares, the primary response first and the elected +content first within each, and those choices order the page and decide nothing else. A type is +rendered once per direction it travels, through its `ModelShape` per lifecycle +([TypePlan](plan/types.md#modelshape--visibility-projection)): a request shape lists what a caller +sends and a response shape what it receives, each member's requiredness read in that direction, so +a write-only property never appears in a response's table, nor a read-only one in a request's. + +## Routes and links + +Every docs target needs URLs, links between pages, an answer for two entities wanting one URL, and +a way to show one entity across API versions. Each is decided by data, never by a renderer: + +- **A catalog first, URLs second.** The emitter builds a catalog — services, groups, operations, + and the types that warrant a page — holding no URL. A route plan then derives every URL from the + catalog alone, through templates in the docs emitter's options, one per kind of page + (`"/{service}/{group}/{operation}"`). A slug is presentation (INV3), rendered from names by the + naming policy and marked `Inferred` where a default derived it. +- **A total route plan.** Every `OpID` and every type with a page gets a route, and every `PropID` + an anchor on its type's page. Anything left without one is a coded diagnostic, and a recursive + type is linked by its ID, never inlined to a depth and cut. +- **Collisions are refused.** Two entities whose routes resolve to one URL are an error + diagnostic naming both, never a page that overwrites another and never a page dropped. The check + runs on the URLs the templates produce, over every kind of route in one table, so no template can + reintroduce a collision the catalog did not have. +- **Links are requests.** A renderer never formats a URL. It writes a link request, an entity's ID + with the projection it wants (a page, an anchor, the agent document of one target), and the + route plan answers it. A spec's [`Docs.Refs`](../ir/annotations.md#references-in-prose) resolve + the same way, so a link follows its target wherever its route moves. +- **Versions keep identity.** The emitter receives one version's snapshot ([contract + rules](contract/emitter.md#contract-rules-the-type-system-and-tests-enforce)), and an entity's + page is identified by its ID, never by the version, so a version switcher links one entity + across versions. The engine computes each snapshot's route table, a pure function of the snapshot + and the options, and hands every snapshot's docs run the tables of all of them in + `EmitterInput.Routes`, for a target whose `Capabilities.CrossVersion` is set. A link to an entity + a version lacks resolves to that version's page saying so. Slugs may differ between versions; IDs + do not. + +## Documents for agents + +Beside its pages, the docs emitter writes the same content as plain Markdown for coding agents, from +the same `Doc + Plan` and through the same route plan, so an agent document and a page never +disagree about a URL: + +- **One document per operation**, holding its method and path, parameters and schemas as nested + lists, examples, code samples and links, as one aggregate document and one per SDK target, so an + agent reads only the language it writes. +- **`llms.txt` indexes** at site, service, version and target scope, each a Markdown index linking + the agent documents of its scope rather than the pages. +- **Facts, not guesses.** An operation is marked safe, or safe to retry, from `OpPlan.Effects` + ([OpPlan](plan/operations.md)). A warning that it destroys something comes only from a declared + fact, a hint or a promoted extension, since a method cannot say it: `DELETE` and `PUT` are + idempotent and may destroy. +- **Spec text is escaped.** A heading or a list item built from a summary or a description is + escaped for Markdown, and raw HTML in a description is rendered as text, never passed through. + +## Samples come from the SDK emitters + +How a call is spelled in one language — the client accessor, the method name, the arguments, a +model's constructor — is that SDK emitter's refine output, and an emitter never sees another +(INV1). A docs emitter that spelled calls itself would re-derive them and could disagree with the +SDK it documents. So each SDK emitter returns a usage sample per operation in +`EmitterOutput.Samples`, and the engine, which runs the SDK targets first, hands the docs emitter +every target's samples in `EmitterInput.Samples`. A sample a target cannot render is a coded +diagnostic of that target's, never an empty code block on the page. Each sample's values are the +plan's examples (below). + +## One set of examples + +READMEs, samples, generated tests and mock responses all need example values, and each would +synthesize its own where the document declares none, so the docs would show values the SDK's own +tests never sent, and every regeneration could change them. So the plan holds one set, +`Plan.Examples`, keyed by the ID of each type, property, parameter and operation: + +- **Declared first.** An entity's `Examples` are its set, labelled authored in the docs. +- **Synthesized only where none is declared**, under `plan.Policy.Examples`: a value valid against + the entity's type and constraints, bounded in depth and in the items it lists, marked `Inferred` + and labelled generated. Each choice the synthesizer makes, such as which enum member or which + variant, is seeded from the entity's correlation key, so the value is the same on every + regeneration until the entity's schema or the synthesizer's step version changes. A value it + cannot make valid, as a `pattern` it cannot satisfy, is reported and left out, never emitted + invalid. +- **Shared** by the docs emitter, the samples, each SDK's tests and the mock server. + +The manifest need not persist them: a seeded synthesizer reproduces each value from its inputs, and +a value that changes because its schema did is one that had to. + +## Documentation extensions + +Documents carry presentation as vendor extensions: a logo, a tag's display name, groups of tags, +code samples on an operation, a marker on an internal operation or property. They stay verbatim in +`Unmodeled`, and the docs emitter reads those its options name, as the [embedded +reader](contract/embedded-policy.md) reads policy: a key to a presentation field, the value decoded +strictly and a wrong shape reported, the entry left in place, and an empty default mapping, since +each key is one tool's spelling. A code sample a document declares is shown as authored beside the +samples the SDK emitters return. An internal marker is the filter pass's to read (architecture.md +§2.2), never the docs emitter's: hiding an operation in the docs alone would document an SDK +different from the one generated. A search index over the pages, keyed by ID, is an optional +artifact of the same emitter. + +## A command-line target + +A CLI is a registry entry over `Doc + Plan`, planned as an extension point beyond the milestones +([milestones](milestones.md)) rather than a milestone target, since nothing in the first targets +depends on it. What it needs is almost all IR and plan facts: + +| CLI behavior | Read from | +|---|---| +| `--dry-run` on a call that changes something; no confirmation on a safe one | `OpPlan.Effects` | +| A confirmation before a destructive call | a declared fact only: a hint or a promoted extension | +| Streamed output for a list | `OpPlan.Page` | +| Choices and defaults on a flag | `Enum`, `Parameter.Default`, `Property.Default` | +| Body fields as flags, with the path to put each back | the request `Payload`'s model, by `PropID` | +| Flags that exclude or require each other | the branches of a request body's `Union` | +| A flag whose value is never echoed | `Property.Secret`, `TypeCommon.Sensitive` | +| Completing an ID from a list call | `ResourceInfo` and its collection operations | + +Command names, positional arguments, `@file` inputs and hidden commands are presentation, so they +are [shaping hints](contract/shaping-hints.md) keyed by ID. Flag constraints are derived at refine +from the union's branches by `PropID`, never by a property's leaf name: a leaf name derives no +conflict between `{origin: {host}}` and `{proxy: {host}}`, and lets two branches' `host` merge into +one flag. Flag names collide as identifiers do and are told apart by the same ID-keyed resolution +([naming](refine/naming.md)), so `foo_bar` and a nested `foo.bar` do not both become `--foo-bar`. A +body field nested at any depth, or with `.` in its name, is a flag like any other, and a union +beside properties, or a top-level `anyOf` body, still yields flags, its branches' constraints among +them. diff --git a/docs/emitters/emit/templates.md b/docs/emitters/emit/templates.md index 20810fbe3..e075a709d 100644 --- a/docs/emitters/emit/templates.md +++ b/docs/emitters/emit/templates.md @@ -6,7 +6,7 @@ |---|---|---| | Models, enums, unions, discriminators, operations, clients, serializers | **Typed AST nodes** | Spec-derived *structure*; correctness must be compiler-checked; must be uniform across languages | | Runtime core: HTTP transport, retry loop, backoff+jitter, pagination iterator runtime, a stream reader per framing, the error and response types, auth injectors, logging hooks | **Templates**, one small file each, parameterised by `Policy` | Policy-driven *boilerplate* whose shape does not vary with the spec | -| Package docs, README, per-op usage examples | **Docs emitter** (`docast`, [docs and mocks](../docs-and-mock.md)) | Prose with holes | +| Package docs, README, per-op usage examples | **Docs emitter** (`docast`, [docs and mocks](../docs-and-mock.md)), each usage sample returned by the SDK emitter whose call it spells | Prose with holes; a call is spelled by the target that generates it | A retry loop is the same code for every API; only numbers and header names vary. A typed AST there buys nothing and costs readability. But a template with an `if` that inspects the IR is a design bug diff --git a/docs/emitters/milestones.md b/docs/emitters/milestones.md index 8c9a0ebfe..08ecdca76 100644 --- a/docs/emitters/milestones.md +++ b/docs/emitters/milestones.md @@ -17,12 +17,17 @@ Extends architecture.md §6 milestone 3 (the first emitters) and beyond. 4. **M3d — docs + mock emitters.** Prove reuse of `Doc + Plan` over a second and third target AST, and that docs/mock agree with the SDK by construction. 5. **M4 — write/integrate + surface verification.** Manifest, header-gated pruning, ignore regions, - additive `go/ast` merge, staleness; neutral surface diff with injectable per-language severity. + additive merge (`dst` for Go), regions that follow their entity, README blocks, staleness; + neutral surface diff with injectable per-language severity. 6. **M5 — a further language family (TS or Python).** Proves the plan holds beyond the first targets' statically typed languages: a new target is a new target AST + printer + refine pipeline reusing the shared plan and step contract, forcing **no** IR or plan change. The shared plan first pays back at M3, where three targets read it. +Beyond the milestones, a command-line client is a documented extension point rather than a +scheduled target: a registry entry over `Doc + Plan`, as docs and mocks are, whose inputs and flag +rules [docs and mocks](docs-and-mock.md#a-command-line-target) sets out. No milestone depends on it. + ## What M3 lowers M3a to M3c lower the IR facts the compilers produce, and degrade the rest with a code. This scopes diff --git a/docs/emitters/open-questions.md b/docs/emitters/open-questions.md index 427959a27..7c2135591 100644 --- a/docs/emitters/open-questions.md +++ b/docs/emitters/open-questions.md @@ -19,8 +19,8 @@ RPC/messaging targets are the real test. Revisit when the second-protocol emitter lands (tracks [open question 6](../ir/open-questions.md)). 5. **Manifest granularity for merge.** Whether the entity→symbol map keys at property level (not just - type/op level) to support field-level additive merge, or leaves sub-type merge to the `go/ast` - adapter. Decide when write/integrate (M4) is built. + type/op level) to support field-level additive merge, or leaves sub-type merge to the + merge adapter. Decide when write/integrate (M4) is built. 6. **Surface behavioral channel scope.** Which behavioral changes beyond default-value changes (e.g. a `Retryable` flip, an idempotency change) belong in the separate behavioral channel vs the structural one. Decide when surface verification (M4) is built against real cross-revision diffs. diff --git a/docs/emitters/operations/client.md b/docs/emitters/operations/client.md index f4535ce5e..189869b98 100644 --- a/docs/emitters/operations/client.md +++ b/docs/emitters/operations/client.md @@ -40,3 +40,32 @@ operation whose `Servers` names a server of its own. authenticates per server, and a messaging target reads it. - **emit.** The URL template is expanded at request time from the chosen values; the template and each variable's constants are typed AST, the expansion a runtime helper. + +## Client settings + +A parameter an API's calls all send alike, such as an account in the path or an API version +header, is set once on the client when its service's hint names it (`ShapeHint.ClientParams`, +[shaping hints](../contract/shaping-hints.md)). Nothing makes one a setting by inference. Whether +every operation of a scope takes a parameter changes as the API grows, so a setting the plan chose +would come and go with operations, and each change would move the parameter into or out of every +method's signature: a breaking SDK change caused by an additive spec change. + +- **What a setting fills.** A `ParamKey` fills, in every operation, the parameter at its location + and wire name whose declaration lowers to one wire form with the others: the same root through + its scalar chain ([scalars](../refine/scalars.md)), and the same style and explode. Requiredness, + constraints and docs may differ, and each operation checks its own. A declaration of another wire + form stays a plain input, reported once by refine, naming the operation, since a shared name does + not make two parameters one. +- **How a call reads it.** The client takes the setting when it is built, a call may give its own, + and a call with neither for a required parameter is refused unsent, the error naming it. A + parameter a setting fills is not positional but an optional field of the operation's request type + ([request shaping](../refine/requests.md)), so a call can still name another account. +- **Candidates.** The plan reports, as an info `emitters/plan/client-param-candidate`, each + `ParamKey` that every operation of some group with more than one operation takes in one wire + form, with the number of operations in the document taking it so, so an SDK's author can name + it. On the golden spec (spec @319dc756, morphic @b206dc99), `account_id` in the path is taken by + 2,660 of 3,763 operations, each a string under one of 16 sets of constraints, and `zone_id` by + 698. One shared component parameter was weighed as the test instead and finds a parameter every + operation of a group shares in 52 of the 515 groups with more than one operation, against 465 by + location and name: the spec declares its parameters inline. + diff --git a/docs/emitters/operations/errors.md b/docs/emitters/operations/errors.md index 9f0305299..59e5ab368 100644 --- a/docs/emitters/operations/errors.md +++ b/docs/emitters/operations/errors.md @@ -24,9 +24,9 @@ spells it ([Go](../targets/go/errors.md#ratelimited-in-go), response decoder fills from the entry a response takes. The retry wiring reads `Retryable`/`Throttling` **from the IR fact** and falls back to `Policy.Retry` only where the spec is silent ([precedence](../contract/runtime-policy.md#precedence--declared-ir-facts-win-hard-rule)); -throttling selects the throttling backoff class. The error types are runtime boilerplate ([structure -and boilerplate](../emit/templates.md)): their kinds come from `Policy.Errors`, so a template -renders them and no spec changes them. +throttling selects the throttling backoff, `Retry.Throttled`. The error types are runtime +boilerplate ([structure and boilerplate](../emit/templates.md)): their kinds come from +`Policy.Errors`, so a template renders them and no spec changes them. ## The status tells failures apart @@ -70,7 +70,11 @@ answers](../plan/responses.md#which-entry-answers)): - a `value` body decodes into `Body`, whatever `Type`'s kind, and a `void` or `raw` one leaves it nil, with `RawBody` holding whatever arrived; - a body that does not decode never hides the failure: `Body` stays nil and `BodyErr` holds the - decode error, since the status is what the caller must learn. + decode error, since the status is what the caller must learn; +- a failure an envelope reports in-band, a 2xx whose flag holds the failing value + ([envelopes](../plan/responses.md#envelopes)), takes no entry: it has no `Fault` and no kind, + since its status states neither, its `Body` is what `EnvelopeRead.Failure` decodes, and it is + retried only as its status would be, which no default retries. A body is buffered up to `Policy.Errors`' bound and no further, so an endless one cannot exhaust the caller; `RawBody` holds what was read, and a body cut there is one that does not decode. A union diff --git a/docs/emitters/operations/returns.md b/docs/emitters/operations/returns.md index 35634a209..a340cdec3 100644 --- a/docs/emitters/operations/returns.md +++ b/docs/emitters/operations/returns.md @@ -63,6 +63,14 @@ spelled as the target spells it ([Go](../targets/go/returns.md), A zero value would not do here: Kotlin's `Rule` has none, a pager has no value to zero, and a nil `*Rule` beside a nil error reads as a created rule until the caller's next line dereferences it. +- **An envelope** (`GET /zone` → `{success, errors, messages, result: Zone}`, under an envelope a + configuration names). **plan.** `Return.Type` is `Zone`, and `Return.Envelope` says it sits at + `result` and that `success: false` is a failure ([envelopes](../plan/responses.md#envelopes)). + **refine.** The method returns `Zone`, and a body whose flag says it failed is a failure, never a + `Zone` ([failures](errors.md)). The envelope's other members are not lost: an enveloped operation, + a paginated one aside, has a `WithResponse` method whatever its headers, and its `Response` holds + the envelope whole in `Envelope`, as `EnvelopeRead.Body` lowers, beside `Body`. On a response with + no envelope, `Envelope` is the target's empty value. - **A status the document does not list, and a body that does not decode** (`GET /zone` → `201`, or a `200` holding a proxy's HTML). **plan.** No field: [the plan](../plan/responses.md#which-entry-answers) says which entry answers. **refine.** The response diff --git a/docs/emitters/operations/streaming.md b/docs/emitters/operations/streaming.md index 58f29b4ce..ff7de1201 100644 --- a/docs/emitters/operations/streaming.md +++ b/docs/emitters/operations/streaming.md @@ -26,8 +26,10 @@ decode](../plan/responses.md#a-body-that-does-not-decode)). What a stream yields `Recv`: - an `Element` lowers as a field of its type does ([lowering table](../refine/lowering-table.md)), - whatever its kind. A stream ends after a variant whose `Variant.Event.Terminal` is true, so an - event union needs nothing of its own. Under `FrameMembers` the element decodes from the frame's + whatever its kind. A stream ends after an element holding a variant whose + `Variant.Event.Terminal` is true, any of the several an [inclusive + union](../refine/unions.md#a-value-several-variants-admit) holds included, so an event union + needs nothing of its own. Under `FrameMembers` the element decodes from the frame's `data`, `event`, `id` and `retry`, and otherwise from its `data`. - no `Element` yields each frame's text, unread. - a `bytes` side has no frames: a side the client reads lowers as a file does ([return diff --git a/docs/emitters/packages.md b/docs/emitters/packages.md index f4f70406b..034e7c31d 100644 --- a/docs/emitters/packages.md +++ b/docs/emitters/packages.md @@ -8,7 +8,8 @@ emitters/ ├── contract.go # Emitter, EmitterInput/Output, Artifact, Capabilities, TargetKey, registry ├── plan/ # SHARED, language-neutral — imports ir, ir/irfacts. Build(), Plan, OpPlan, TypePlan, ModelShape │ └── plantest/ # golden snapshots of Plan (language-independent corpus) -├── policy/ # runtime SDK-policy vocabulary + Defaults() + Merge(DeepPartial) (INV10) +├── policy/ # runtime SDK-policy vocabulary + Defaults() + Merge(DeepPartial) + Resolve(layers) (INV10) +├── embed/ # reads the policy and hints a document embeds as vendor extensions (contract/embedded-policy.md) ├── naming/ # SHARED naming engine: acronym/compound tables, the character-name table, │ # reserved-word set type, ID-keyed collision resolver. Casing FUNCTIONS are │ # per-target (naming.Policy impl) @@ -24,7 +25,7 @@ emitters/ │ ├── refine/ # IR+Plan → goast — the ordered lowering pipeline (refine/pipeline.md) │ ├── emit/ # goast → []byte via printer + go/format; templates/ (boilerplate only) │ ├── naming/ # Go reserved words + default acronym table (naming.Policy impl) -│ └── mergeadapter.go# go/ast additive-merge adapter (registers into manifest driver) +│ └── mergeadapter.go# dst additive-merge adapter (registers into manifest driver) ├── kotlin/ # ◀ FIRST TARGETS (TargetKeys "kotlin-jvm" and "kotlin-multiplatform") │ ├── emitter.go # NewJVM() and NewMultiplatform(): one pipeline, two Platform profiles │ ├── platform.go # Platform: the built-in type table, identifier rules, transport @@ -50,7 +51,9 @@ its own, as a compiler's packages do, and none may be exempt, which only tooling is. The entries the first packages add assert: `emitters/golang` and `emitters/kotlin` may each import `emitters`, `emitters/plan`, `emitters/policy`, `emitters/naming`, `emitters/typemap`, `emitters/refine`, `emitters/manifest`, `emitters/verify`, `ir` and `ir/irfacts`; neither may import -`compilers/*`, `engine`, or the other. +`compilers/*`, `engine`, or the other. `emitters/golang` may also import `github.com/dave/dst`, for +its merge adapter, and no other package may +([merging](targets/go/overview.md#merging-and-surface-extraction)). **Switch-completeness tests** live beside `goast` and `ktast` (over each one's `NodeKind`) and beside every refiner (over every `ir.TypeKind`): adding an IR kind fails the test of every dispatch that must handle it (the `assertNever` obligation). diff --git a/docs/emitters/plan/operations.md b/docs/emitters/plan/operations.md index b5b52f8ea..2292e8984 100644 --- a/docs/emitters/plan/operations.md +++ b/docs/emitters/plan/operations.md @@ -15,6 +15,8 @@ type OpPlan struct { Stream *StreamPlan // nil = neither direction streams (docs/ir/interaction-patterns.md) LRO *LROPlan // nil = not long-running Auth []AuthOption // OR-of-ANDs, priority-ordered (../operations/auth.md) + Effects Effects // what calling it may change: declared, else by method + // (policy.md) Bindings BindingView // per-protocol param/request/response view from OpBindings Inferred map[string]ir.Provenance // which fields above were heuristic (INV6) } @@ -40,6 +42,12 @@ type LROPlan struct { // maps ir.LongRunning (docs/ir/interactio ResultPath *ir.PropPath // where the final result lives in the terminal response } +type Effects struct { + Safe bool // asks for no change on the server (RFC 9110 §9.2.1) + Idempotent bool // a second call changes nothing the first did not + // (RFC 9110 §9.2.2); true of every safe operation +} + type AuthOption struct { // one option in the OR-of-ANDs; slice order is PRIORITY order Schemes []ir.SchemeUse // ALL must be satisfied together; empty = "no auth is acceptable" } @@ -55,10 +63,12 @@ lowered late. The content ranking (`json > form > multipart > binary`) is a shar helper (resolves [open question 3](../../ir/open-questions.md)). Some facts oagen's plan carries are not here, by the [plan-or-refine test](plan-or-refine.md). -Whether there is a body, whether the operation is one-way and its idempotency are stated by the IR, -so a refiner reads them there: `Operation.Request`, `Operation.OneWay` and `Operation.Idempotency`, -the last of which also names the idempotency-token parameter. And whether an operation's inputs are -passed positionally or inside an options object differs by language, so it is each target's +Whether there is a body and whether the operation is one-way are stated by the IR, so a refiner +reads them there, `Operation.Request` and `Operation.OneWay`, as it reads the idempotency-token +parameter `Operation.Idempotency` names. Whether the operation is safe or idempotent is `Effects`, +since the IR states it for few operations and a method states it for most ([plan +policy](policy.md#an-operations-effects-come-from-its-method)). And whether an operation's inputs +are passed positionally or inside an options object differs by language, so it is each target's ([request shaping](../refine/requests.md)); the neutral facts it is decided from are already here — `BindingView` says where each input binds, and `Parameter.Required` and `Payload.Required` which are required — so moving the decision out added no field. diff --git a/docs/emitters/plan/overview.md b/docs/emitters/plan/overview.md index e9ff55e13..ca8b82aa9 100644 --- a/docs/emitters/plan/overview.md +++ b/docs/emitters/plan/overview.md @@ -15,15 +15,28 @@ func Build(doc *ir.Document, facts *irfacts.Index, pp Policy) (*Plan, []ir.Diagn type Plan struct { Ops map[ir.OpID]OpPlan // one entry per operation Types map[ir.TypeID]TypePlan // one entry per registry type - Version string // hash(doc content-hash + PlanPolicy); feeds the manifest + Examples map[string][]Example // by the ID of each type, property, parameter and + // operation: its declared examples, else one + // synthesized (../docs-and-mock.md#one-set-of-examples) + Version string // the plan's identity (below), which the plan cache + // and the generation manifest key on +} + +const Revision = 1 // bumped with every change to what Build returns + +type Example struct { + Value ir.Example // the example, as the IR spells one + Authored bool // declared by the document; false = synthesized, Inferred } ``` Every key is an IR stable ID — never `"METHOD /path"`, never a rendered name (INV3; the oagen -fragility, prior-art §1). A spec rename never orphans a plan entry: it is "same ID, new symbol", -a lookup. Every inferred field carries `Inferred` provenance, propagated into diagnostics so it is -auditable and disable-able (INV6). The plan's own diagnostics are coded `emitters/plan/…`, reported -once for every target rather than once by each. +fragility, prior-art §1). A rename of anything an ID is not built from, such as an `operationId` or +a `title`, never orphans a plan entry: it is "same ID, new symbol", a lookup. A rename of what an ID +is built from, a path or a component name, gives the entity a new ID, and only its correlation key +(#679) carries it across the revision. Every inferred field carries `Inferred` provenance, +propagated into diagnostics so it is auditable and disable-able (INV6). The plan's own diagnostics +are coded `emitters/plan/…`, reported once for every target rather than once by each. The plan **classifies; it does not pick target types and it never cases anything.** It records what it derives from the IR ("this union is discriminated, and its tag values select the variants with @@ -37,3 +50,23 @@ derivation reads a type's contents, such as the property a page's items are foun them through `TypePlan.Alias` ([decorators](../refine/decorators.md)), and every consumer lowers a reference it reads through `Alias`, the plan's as the IR's, so no field holds a reference already lowered beside others that are not. + +## The plan's version + +`Version` is a SHA-256 hash of three things: the document's digest, `Revision`, and the policy's +encoding as `encoding/json/v2` writes it under `json.Deterministic(true)`. The digest is SHA-256 +over the document's canonical JSON (`MarshalJSONTo`) with `sources` left out, which `Build` never +reads, so a spec reformatted into the same IR keeps its plan's version. Two builds therefore share +a `Version` exactly when they share a document, a policy and the code that built them, and the plan +cache and the generation manifest ([write / integrate](../write-integrate.md)) can trust it. + +- **The policy is data** ([plan policy](policy.md#the-policy-is-data)), so its encoding is what it + says. A policy holding a function would encode to nothing stable: json/v2 refuses the field, its + printed form is an address that changes every run, and leaving it out gives two policies that pick + different responses one version. +- **`Revision` stands for the code.** A change to `Build` that changes its output for any document + bumps it, and the plan's golden corpus holds the bump: a golden that changes at an unmoved + `Revision` fails, as a shape change does at an unmoved `IRVersion` + ([IR version](../../ir/ir-version.md)). Without it, a fix to the plan would leave every cached + plan current. + diff --git a/docs/emitters/plan/plan-or-refine.md b/docs/emitters/plan/plan-or-refine.md index 028b8812b..3bcf198ba 100644 --- a/docs/emitters/plan/plan-or-refine.md +++ b/docs/emitters/plan/plan-or-refine.md @@ -31,24 +31,28 @@ of the artifact that uses it — how an untagged union's variant is recognized o The procedure decided every existing field: 1. **Choices under `plan.Policy`** stay: `PrimaryResponse`, `RequestContent` and the content each - `ResponsePlan` elects, the unwrap behind `Return.Type`, `Open`, and `PagePlan`, an inferred one - included. + `ResponsePlan` elects, the unwrap behind `Return.Type`, `Open` on a union and on an enum, + `Effects`, which a method sets where the IR states nothing, `Examples`, synthesized where none + is declared, and `PagePlan`, an inferred one included. 2. **Derivations several artifacts expose** stay: `Return`, `Others`, `Errors`, `Auth`, `Bindings`, `Stream` and `LRO`; and `Flattened`, `Required`, `Undeclared`, `Unflattened`, `Union`, `Discrim`, - `Scalar`, `Shapes`, `Alias` and `Reach`. `Required`, `Undeclared` and `Unflattened` are read - along the flattening walk ([TypePlan](types.md)), `Alias` classifies a decorator over several of - its fields and resolves its chain ([decorators](../refine/decorators.md)), and `Reach` walks - every operation ([output units](../refine/output-units.md)); SDKs and docs that disagreed on any - of them would mark a member optional, name a field's type or place a type differently. -3. **Copies of what the IR states** are gone: whether there is a body, one-way-ness and idempotency - ([OpPlan](operations.md)), a model's `Base`, `Implements` and `Mixins`, an enum's openness, value - type, flags and fallback member, a union's variant list, and the operation-group tree - ([BindingView](bindings.md)). `Union` replaces a classification that copied the variant list and - a second discriminator beside the tag mode it derived, and `Scalar` replaces a bare root that - left the constraints it merged nowhere. `NeedShapes` is gone too: it listed again the lifecycles - `Shapes` is keyed by ([ModelShape](types.md#modelshape--visibility-projection)). A client's - servers are stated by the IR as well, so they get no field ([the - client](../operations/client.md)). + `Inclusive`, `Scalar`, `Shapes`, `Alias` and `Reach`. `Required`, `Undeclared` and `Unflattened` + are read along the flattening walk ([TypePlan](types.md)), `Inclusive` compares every pair of a + union's variants ([TypePlan](types.md#variants-that-can-share-a-value)), `Alias` classifies a + decorator over several of its fields and resolves its chain + ([decorators](../refine/decorators.md)), and `Reach` walks every operation ([output + units](../refine/output-units.md)); SDKs and docs that disagreed on any of them would mark a + member optional, name a field's type, hold one variant where another holds several, or place a + type differently. +3. **Copies of what the IR states** are gone: whether there is a body, one-way-ness and the + idempotency token ([OpPlan](operations.md)), a model's `Base`, `Implements` and `Mixins`, an + enum's declared openness, value type, flags and fallback member, a union's variant list, and the + operation-group tree ([BindingView](bindings.md)). `Union` replaces a classification that copied + the variant list and a second discriminator beside the tag mode it derived, and `Scalar` replaces + a bare root that left the constraints it merged nowhere. `NeedShapes` is gone too: it listed + again the lifecycles `Shapes` is keyed by + ([ModelShape](types.md#modelshape--visibility-projection)). A client's servers are stated by the + IR as well, so they get no field ([the client](../operations/client.md)). 4. **The parameter-passing shape** fails the first test and moved to refine ([request shaping](../refine/requests.md)): a dynamic target's keyword arguments are additive as they stand, where a positional parameter added to a Go or Kotlin function breaks its callers, so the diff --git a/docs/emitters/plan/policy.md b/docs/emitters/plan/policy.md index 8cbbdfd0a..4c3c736bf 100644 --- a/docs/emitters/plan/policy.md +++ b/docs/emitters/plan/policy.md @@ -5,35 +5,67 @@ Everything the plan *infers* rather than reads is a knob here; every inferred ou ```go type Policy struct { - ContentRanking []string // default: json > form > multipart > binary - PrimaryResponse func(*ir.Operation) *ir.Response // default: lowest 2xx, else the - // unconditional one, else the Default - EnvelopeDetect EnvelopePolicy // off unless explicitly enabled - Framings map[string]Framing // media type or +suffix → how it frames a stream (streams.md) - FrameMembers bool // default on: an SSE schema naming only frame - // fields describes the frame (streams.md) - OpenUnions bool // default on: a union a client reads keeps a - // variant it does not know (below) + ContentRanking []string // default: json > form > multipart > binary + PrimaryResponse []PrimaryRule // default: lowest_2xx, unconditional, default; the first + // rule an operation meets picks its primary (below) + DefaultFails bool // default on: a Default no rule takes is the catch-all + // of Errors (below) + Envelopes []Envelope // default none: what unwraps a success body (below) + Framings map[string]Framing // media type or +suffix → how it frames a stream + // (streams.md) + FrameMembers bool // default on: an SSE schema naming only frame + // fields describes the frame (streams.md) + OpenUnions bool // default on: a union a client reads keeps a + // variant it does not know (below) + OpenEnums bool // default on: an enum a client reads keeps a value + // it does not list (below) + ExclusiveMatch string // "first" (default) | "only": what a oneOf value two + // variants admit decodes to (../refine/unions.md) + MethodSemantics bool // default on: an operation stating no effects takes + // its method's (below) + Examples ExamplePolicy // default on, depth 3, 2 items: what may be synthesized + // where no example is declared (../docs-and-mock.md) } + +type ExamplePolicy struct { + Synthesize bool // off leaves an entity with none declared without one + MaxDepth int // how deep a synthesized value nests + MaxItems int // how many items a synthesized list or map holds +} + +type PrimaryRule string // "lowest_2xx" | "unconditional" | "default" ``` A test asserts that disabling a heuristic removes the corresponding `Inferred` plan field — the guard against a heuristic silently defaulting on and quietly making the plan non-neutral. +## The policy is data + +Every field is a value from a closed vocabulary the plan package defines, and none is a function. +So a policy encodes, compares and hashes as what it says, and `Plan.Version` hashes it +([the plan](overview.md#the-plans-version)). A function has no bytes to hash: its printed form is +an address that changes every run, and skipping it gives two policies that pick different responses +one version. A heuristic the vocabulary lacks is added to it, in this package and at a new +`plan.Revision`, so a change of behaviour always changes the version. A strategy registered by name +and version from outside was weighed and refused: changing its code without its version would be a +silent cache hit, which is the failure this rule removes. So was a caller-supplied ID hashed in the +policy's place, which fails the same way when the caller forgets it. + ## The default response is classed here The IR states an operation's `Default` — OpenAPI's `default` — as neither a success nor a failure, since it answers every status the others do not name ([responses](../../ir/responses.md)). -`PrimaryResponse` reads the whole operation for that reason, and its default takes the lowest 2xx -`Response`, an exact status before a range that starts there, else one with no condition, which is -an RPC's one response, or the `Default` where the operation declares neither, since a `default` -answering every status is then the operation's only description of a success. A `Default` that is -not the primary response is the catch-all of `Errors`, with its condition. Either use is marked -`Inferred`, and a policy may take it as neither or both. Disabling the heuristic leaves an operation -with no success response `void` ([body shapes](responses.md#body-shapes)) and its `Default` in no -plan field. `Streaming` summarizes only the streams an operation's success responses declare, so no -`StreamDetail` covers a `Default`, and a primary one streams only where a content of it declares an -`Item` ([streams](streams.md)). +`PrimaryResponse` reads the whole operation for that reason, rule by rule, and the first rule the +operation meets picks: `lowest_2xx` the lowest 2xx `Response`, an exact status before a range that +starts there; `unconditional` one with no condition, which is an RPC's one response; and `default` +the `Default`, which is the only description of a success an operation declaring neither has. A +`Default` that is not the primary response is the catch-all of `Errors`, with its condition, under +`DefaultFails`. Either use is marked `Inferred`, and a policy may take it as neither or both. +Leaving out the `default` rule and `DefaultFails` leaves an operation with no success response +`void` ([body shapes](responses.md#body-shapes)) and its `Default` in no plan field. `Streaming` +summarizes only the streams an operation's success responses declare, so no `StreamDetail` covers a +`Default`, and a primary one streams only where a content of it declares an `Item` +([streams](streams.md)). ## A union a client reads is open @@ -48,6 +80,83 @@ has no field saying a union is open or closed (#664); once it does, a declared a plan records the choice so that docs and a mock agree with the SDK ([docs and mocks](../docs-and-mock.md)). +## An enum a client reads is open + +The reason unions are open holds for an enum: an API that adds a member must not break every SDK +already deployed on the first response carrying it. So under `OpenEnums` an enum is `Open` wherever +`Facts.Usage` marks it `Output`, marked `Inferred`, and a value it does not list is kept rather +than refused ([enums](../refine/enums.md)). An enum only ever written stays as the IR says, and +turning the policy off leaves every enum as the IR says. + +This does not override the declared `Enum.Closed`, which states that the members the document +lists are all it admits. The SDK honours that in what it sends: its constants name the listed +members alone. `Closed` says nothing of what a later version of the API will list, and that is what +a client reading a response has to survive. A document declaring an enum open, through a promoted +`x-extensible-enum` among others, makes it open in both directions. On the golden spec (spec +@319dc756, morphic @b206dc99) all 8,314 enums are closed, since an OpenAPI `enum` is a closed value +set, so without the policy each one a response carries would fail to decode the first time the API +added a member. + +## An operation's effects come from its method + +`OpPlan.Effects` ([OpPlan](operations.md)) says whether calling an operation is safe and whether it +is idempotent. A declared `Operation.Idempotency` decides, its token kind counting as idempotent +since a retry sends the token it first sent. Where the IR states none, `MethodSemantics` takes them +from the binding's method as RFC 9110 §9.2 defines them, marked `Inferred`: `GET`, `HEAD`, +`OPTIONS` and `TRACE` are safe, and those with `PUT` and `DELETE` are idempotent. `POST` and +`PATCH` state neither, and an operation no HTTP binding carries takes neither. + +A method says nothing of whether an operation destroys something: `DELETE` and `PUT` are +idempotent and may be destructive, so nothing derives a confirmation from `Effects`. On the golden +spec no operation declares its idempotency, and of its 3,763 operations 1,841 use a safe method and +2,693 an idempotent one. Without the policy no operation would be known idempotent, so no request +that got no response would ever be sent again. + +## Envelopes + +An envelope wraps an API's success bodies in members of its own, such as a failure flag, messages +and page information, beside the one member a caller wants. An SDK that returns the envelope makes +every call site dig. One that unwraps it has to know where the value and the failure signal live, +so a configuration names its envelopes, and the default names none. + +```go +type Envelope struct { + Name string // recorded in each Inferred mark this envelope sets + Bases []ir.TypeID // matches a body whose model reaches one of these through Base or Mixins + Members []string // matches a body whose model declares every one of these wire names + Value []string // the wire-name path to what the reader gets in the body's place + Flag []string // the path to the member saying whether the call succeeded; nil = none + Fails *ir.Value // the Flag value meaning the call failed; set exactly when Flag is + Except []ir.OpID // operations whose bodies this envelope never matches +} +``` + +A body matches an `Envelope` when its type, read through `TypePlan.Alias`, is a model that reaches +one of `Bases` through `Base` or `Mixins`, or declares every name in `Members`, read through +`Flattened`. Either way it must also declare the members `Value` and `Flag` name. A body takes the +first envelope it matches, and none of them on an operation its `Except` lists. +[Responses](responses.md#envelopes) says how a matched body is read. Each field the match sets is +marked `Inferred` with the envelope's `Name`. An `Except` or `Bases` entry naming nothing in the +document is reported, as is a `Flag` without `Fails`. + +A body is matched by its members as well as by its composition, because composition alone misses +real envelopes. On the golden spec (spec @319dc756, morphic @b206dc99), the primary bodies of 2,665 +of 3,763 operations declare `success`, `errors`, `messages` and `result`. 2,238 of them reach a +named component declaring the `success` flag through `Base` or `Mixins`, 113 distinct components in +all, since products declare their own; the other 427 declare it inline, and no `Bases` list reaches +them. One `Members` entry matches all 2,665, and since it names every member, a body that merely has +a `result` is never unwrapped by accident: + +```go +plan.Envelope{ + Name: "result", + Members: []string{"success", "errors", "messages", "result"}, + Value: []string{"result"}, + Flag: []string{"success"}, + Fails: &ir.Value{Kind: ir.ValueBool, Bool: false}, +} +``` + ## Framings Framings state what a media type means, not a guess about an API, so the table marks nothing diff --git a/docs/emitters/plan/responses.md b/docs/emitters/plan/responses.md index bd9d3a2fa..5ec046dcf 100644 --- a/docs/emitters/plan/responses.md +++ b/docs/emitters/plan/responses.md @@ -7,10 +7,23 @@ declares, read by one `ResponsePlan`. // ResponsePlan is how one response reaches its reader. A success and a failure are read by the // same rules, so the election, the shapes, the codec and the headers have one home. type ResponsePlan struct { - Shape BodyShape // value | void | raw | stream - Content *ir.Content // the content elected from the payload; nil = nothing to read - Type *ir.TypeRef // what a value body decodes into, of any kind; nil unless Shape is value - Headers []ir.Header // the headers the response declares, each whole (below) + Shape BodyShape // value | void | raw | stream + Content *ir.Content // the content elected from the payload; nil = nothing to read + Type *ir.TypeRef // what a value body gives its reader, of any kind; nil unless + // Shape is value + Envelope *EnvelopeRead // how an enveloped body is read; nil = the body decodes into + // Type (below) + Headers []ir.Header // the headers the response declares, each whole (below) +} + +// EnvelopeRead is a body one of plan.Policy.Envelopes matched (policy.md), as it is read here. +type EnvelopeRead struct { + Name string // the Envelope that matched + Body ir.TypeRef // what the body decodes into: the elected content's type + Value *ir.PropPath // where Type sits inside Body; nil on a pager's page, read whole + Flag *ir.PropPath // the member saying whether the call succeeded; nil = none + Fails *ir.Value // the Flag value meaning it failed + Failure *ir.TypeRef // what a failing body decodes into; nil = the target's untyped JSON } // BodyShape is how a body reaches its reader, a small closed enum: @@ -68,12 +81,36 @@ in a second channel. The first rule that applies decides: plan](overview.md), [decorators](../refine/decorators.md)), so an anonymous `allOf` decorator around a model returns the model. Its `Nullable` is the content's, and a nullable return takes the target's nullable form ([optionality](../refine/models.md#optionality--nullability--presence-inv8)). -Unwrapping an envelope, which `plan.Policy` may enable ([plan policy](policy.md)), would return a -value found inside the content instead; what it records and where is #695's. Which codec reads a +An envelope, which `plan.Policy` may name ([plan policy](policy.md#envelopes)), gives the reader a +value found inside the content instead, as [envelopes](#envelopes) says. Which codec reads a media type is the target's, and a media type it has none for is a coded degrade of that target's that reads the body as `raw`. Every body of every document takes one of the four, since the rules end in a default; [Return shapes](../operations/returns.md) traces each through the first targets. +## Envelopes + +A `value` body that one of `plan.Policy.Envelopes` matches ([plan policy](policy.md#envelopes)) is +read through it, and `Envelope` records how, marked `Inferred` with the envelope's name: + +- **The value.** The body decodes into `Envelope.Body`, and the reader gets the member at `Value`: + `Type` is that member's type, so `Return.Type` is the `Zone` inside `{success, errors, messages, + result: Zone}`. The rest of the envelope stays reachable through the response + ([return shapes](../operations/returns.md)). A paginated operation's page is read whole, as any + page is: `Value` is nil, `Type` stays the page, and its items are where `Pagination.Items` says. +- **A failure in-band.** Where `Flag` is set, the decoder reads that member before decoding the + body, as a discriminator's tag is read ([unions](../refine/unions.md#discriminators)). A body + whose flag holds `Fails` is a failure despite its 2xx status, and decodes into `Failure`: the + `Type` of the first entry of `Errors` whose body the same envelope matches, or the target's + untyped JSON where none does. It is never decoded as `Body`, since an API whose success envelope + declares `success: true` alone sends a failure no success type admits + ([failures](../operations/errors.md#every-failure-has-one-place)). On the golden spec, 2,321 of + the 2,665 enveloped operations declare an error case the envelope matches, 2,225 of them only + one such type. +- **Only a success.** A failure is read whole, so an entry of `Errors` has no `Envelope`, and a + body that does not match is read as it would be with no envelope configured. + +An entry of `Others` is read by the same rules, so `AsReturn` compares the types its reader gets. + ## Headers `Headers` are what the response declares, each whole, so its name, its type and the `Style` and diff --git a/docs/emitters/plan/types.md b/docs/emitters/plan/types.md index 72afbe89e..c927aac5a 100644 --- a/docs/emitters/plan/types.md +++ b/docs/emitters/plan/types.md @@ -12,8 +12,11 @@ type TypePlan struct { // cannot hold: each lowered beside the properties, or degraded Union UnionShape // a union's wire shape, one of four (../refine/unions.md); "" = not a union Discrim *DiscrimFacts // a union's or hierarchy's discriminator, as variant IDs - Open bool // a union or polymorphic base keeps a value no variant - // recognizes rather than refusing it (policy.md, ../refine/unions.md) + Open bool // a union, polymorphic base or enum keeps a value no variant or + // member recognizes rather than refusing it (policy.md, + // ../refine/unions.md, ../refine/enums.md) + Inclusive bool // an any_of union two of whose variants can admit one value, + // holding every variant a value admits (below); false otherwise Scalar *ScalarChain // a scalar's Base chain, walked once; nil = not a scalar Shapes map[ir.Lifecycle]ModelShape // the wire shapes the document exercises (below) Alias ir.TypeID // the type every reference to this one is emitted as, this @@ -69,6 +72,38 @@ to it. A conjunct that is no model, as `Tags` is in `allOf: [{$ref: Pet}, {$ref: `Tags` is a list, is in `Unflattened` as the entry names it: the declaration over `Flattened` does not hold what it admits, so each refiner lowers it or degrades with a code of its own. +## Variants that can share a value + +`Inclusive` is true on an `any_of` union when two of its variants can admit one value, and false on +every other shape. What admitting a value means is in +[unions](../refine/unions.md#a-value-several-variants-admit). The plan proves two variants disjoint, +admitting no value in common, by these rules, and a pair none of them separates can share one: + +- **Kinds.** Their JSON kinds have none in common. A variant's kinds are its type's: object for a + model or a map, array for a list, a tuple or a `Positional` model, its wire kind for a primitive, + its values' for an enum or a literal, its variants' for a union, its root's for a scalar, its + base's for a model that only stands over one, and every kind for `any`. A nullable reference adds + `null`, and an integer is a number too. A variant's own `null` is the union's + ([unions](../refine/unions.md#a-null-a-variant-admits)), so the rules leave it out, and a variant + admitting only `null` takes no part. +- **Values.** Each admits finitely many values, as a closed enum, a literal or a union of them does, + and no value is in both, compared as JSON values. +- **Members.** Both are models laid out by name, read through `Flattened`, and one requires a member + the other either does not declare and is closed, or declares at a position disjoint from it by + these rules. A pair already being compared counts as sharing, so a cycle of models ends the walk. + +The rules are sound, not complete. A pair they leave sharing may still admit no common value, as a +`date` and a `uuid` string do, and is then inclusive: it holds every value correctly, though more +loosely than a closed sum would. `Constraints` take no part, since admitting a value leaves them to +validation, so two variants that differ only in a bound do share values. + +On the golden spec (spec @319dc756, morphic @b206dc99), 515 unions are untagged, not exclusive, and +have two or more variants. These rules prove 419 of them disjoint, and those lower as a closed sum. +Of the 96 left, 72 have two variants that both admit an object, such as the 16 identity providers +an Access listing returns. Each refines one base whose `type` enum lists all 16, and each `config` +is open with no required member, so nothing separates them: one provider's value is admitted by +every other whose `config` declares none of its members at another type. + ## ModelShape — visibility projection One logical model produces N wire shapes across the lifecycle. `ModelShape(model, lifecycle)` is a diff --git a/docs/emitters/refine/enums.md b/docs/emitters/refine/enums.md index 446e2bf1f..abd9cedc1 100644 --- a/docs/emitters/refine/enums.md +++ b/docs/emitters/refine/enums.md @@ -7,3 +7,22 @@ a closed and a flags enum in forms of its own ([Go](../targets/go/types.md#enums the canonical name for serialization. The open/closed bit, value type, member values, and flags bit all survived from the IR to this point — Kiota's string-only-closed-enum mistake is designed out because the fact reached refine. + +An enum `TypePlan.Open` marks ([plan policy](../plan/policy.md#an-enum-a-client-reads-is-open)) +lowers to the target's open form whatever `Closed` says, so a response carrying a member a later +version added decodes and keeps its value. Its constants still name only the members the document +lists, which is all `Closed` asks of what the SDK sends. + +## A member on the wire + +A member's wire form is its `WireName` where it has one, else its `Value` as the JSON kind its +enum's `ValueType` has: a string for a `string`, `date` or `uuid` enum, a number for an `integer`, +`number` or `decimal` one, a boolean for a `bool` one. Every target writes that form and decodes by +comparing a value with the members as JSON compares values, which is what an `enum` keyword +compares, whatever `format` says: numbers by value, so `0.1` and `0.10` are one member, and +everything else by its text. A target whose enum form holds only some value types lowers the others +over the wire form ([Go](../targets/go/types.md#enums), [Kotlin](../targets/kotlin/types.md#enums)), +never over a form that would change the JSON kind, as a `bool` enum written as the string `"true"` +would. On the golden spec (spec @319dc756, morphic @b206dc99) 1,069 enums are `bool`, 35 `integer` +and 348 `number`, beside 6,862 `string` ones. + diff --git a/docs/emitters/refine/lowering-table.md b/docs/emitters/refine/lowering-table.md index 651a3173b..21ba2e37c 100644 --- a/docs/emitters/refine/lowering-table.md +++ b/docs/emitters/refine/lowering-table.md @@ -23,8 +23,8 @@ IR bug (INV9), not a compiler change. | anonymous decorator: a `Model` with only a `Base`, or a `Scalar` with neither `Constraints` nor `Encoding`, beside annotations ([decorators](decorators.md)) | — | | `Model` whose `Base` is no model, not a decorator | — | | `AdditionalProps` / `Additional` | pattern channel | -| `Union` (four wire shapes, [unions](unions.md)) | `WireName`/`WireID` per variant; an unknown variant's value | -| `Enum` open/closed/flags | member `WireName` | +| `Union` (four wire shapes, [unions](unions.md)) | `WireName`/`WireID` per variant; an unknown variant's value; every variant an inclusive union's value admits | +| `Enum` open/closed/flags | member `WireName`; each member's JSON kind, compared by value ([a member on the wire](enums.md#a-member-on-the-wire)) | | `List`/`MapT`/`Tuple` | packed/expanded | | `Literal` | — | | `External` | — | diff --git a/docs/emitters/refine/naming.md b/docs/emitters/refine/naming.md index b4c047d73..91439e92e 100644 --- a/docs/emitters/refine/naming.md +++ b/docs/emitters/refine/naming.md @@ -38,12 +38,43 @@ there is no lossy runtime re-splitting of a cased string (oagen's `splitWords` c `EscapeReservedWords` runs after casing and **never touches `WireName`/`WireID`** — struct tags are built from the wire channel, so wire correctness never depends on the rendered identifier ([T-4](../testing/mandatory-tests.md)). Collisions are resolved deterministically in `Ctx.Syms`, -which is keyed by IR ID: adding one endpoint never renames an existing method (the oagen -collision-cascade counterexample; INV3). Anonymous hoisted types get an emitter-chosen name from -`Naming.Hint`. Per-service presentation renames (`Service.Renames`) change how a shape is presented -in a service without changing its `TypeID` or its own `Naming`. Enum members are keyed by -`EnumMember.ID`, which is derived from the value a member holds, so two members whose names render -alike are two entries to disambiguate, and reordering members of distinct values renames none. +which is keyed by IR ID, and against the names the prior generation shipped, so adding one endpoint +never renames an existing method (the oagen collision-cascade counterexample; INV3; +[collisions](#collisions-and-the-names-a-generation-shipped)). Anonymous hoisted types get an +emitter-chosen name from `Naming.Hint`. Per-service presentation renames (`Service.Renames`) change +how a shape is presented in a service without changing its `TypeID` or its own `Naming`. Enum +members are keyed by `EnumMember.ID`, which is derived from the value a member holds, so two members +whose names render alike are two entries to disambiguate, and reordering members of distinct values +renames none. + +## Collisions, and the names a generation shipped + +Names that render alike in one scope collide, and `Ctx.Syms` tells them apart. Keying by ID and +being deterministic are both needed and neither is enough: no rule reading only the current document +can tell which of two colliding operations shipped first, so any such rule can rename the incumbent +when a newcomer joins. A renamer that renames every member of a collision from its own identity is +deterministic and order-independent, and still renames `get` to `zonesGetZone` the day +`zonesGetZoneSettings` joins its group, a breaking SDK change. Only the prior generation knows the +incumbent, so the prior manifest ([write / +integrate](../write-integrate.md#what-the-manifest-records)) is an input to the rule: + +1. **A shipped name keeps its symbol.** An entity whose prior entry has its ID and its correlation + key, in the same unit, with the same `Base`, takes the entry's `Symbol` again, a suffix it was + given included. Two such symbols were apart in the prior generation and stay apart. +2. **Every other member, in ID order**, takes its rendered name where nothing in the scope holds it, + else that name with the lowest numeric suffix, `2` up, that is free. + +Only the choice is pinned, never the name: an entity whose own rendered name changed, as a renamed +`operationId` does, gets a new `Base`, so rule 1 does not hold for it and it is named afresh. An ID +alone pins nothing, since an ID built from an array index can name another entity in the next +revision (#678): the correlation key must agree too. On a first generation, or one run without its +prior manifest, rule 2 decides alone; ID order is independent of declaration order, so the two-order +oracle holds it. What the promise rests on is stated with it: a name stays only while its entity's +ID and correlation key survive the revision, and an ID derived from a pointer changes when the +pointer does (#419), until the correlation key carries it across. + +The prior manifest is an input like any other, so a generation is still a pure function of its +inputs ([contract rules](../contract/emitter.md#contract-rules-the-type-system-and-tests-enforce)). ## The source spelling, for letter case only diff --git a/docs/emitters/refine/pipeline.md b/docs/emitters/refine/pipeline.md index 8d2dca0e8..da15ad218 100644 --- a/docs/emitters/refine/pipeline.md +++ b/docs/emitters/refine/pipeline.md @@ -21,6 +21,9 @@ type Refiner interface { type Step interface { Name() string // stable, for diagnostics + golden step-trace tests + Version() int // bumped with every change to what it builds for some input; + // the manifest records it + // (../verification.md#during-steps-carry-versions) Apply(*Ctx) []ir.Diagnostic // appends typed nodes to the target AST under construction; idempotent } @@ -29,10 +32,17 @@ type Ctx struct { Facts *irfacts.Index // read-only derived facts, the engine's one index (../contract/emitter.md) Plan *plan.Plan // read-only shared decisions Types typemap.Table // the merged type table (../contract/type-table.md), built from Options - Policy policy.Policy // read-only SDK runtime policy (../contract/runtime-policy.md) + Policy policy.Resolved // read-only SDK runtime policy, resolved per operation + // (../contract/runtime-policy.md#layers-and-operations) Naming naming.Policy // the target's casing engine over its options' naming tables // (INV4; naming.md) Hints map[emitter.EntityID]emitter.ShapeHint + Prior *emitter.PriorState // read-only prior generation, whose shipped names naming keeps + // (naming.md); nil on a first run + Samples map[emitter.TargetKey]map[ir.OpID]emitter.Sample // read-only, a docs target's: each + // SDK target's samples (../docs-and-mock.md) + Routes map[string]emitter.RouteTable // read-only, a cross-version target's: every + // snapshot's routes (../docs-and-mock.md#routes-and-links) Options emitter.TargetOptions // decoded options (../contract/options.md); a step asserts its own target's type Out TargetAST // the per-language AST being built (e.g. *goast.Module) Syms *SymbolTable // ID-keyed; collision resolution never touches IR identity diff --git a/docs/emitters/refine/requests.md b/docs/emitters/refine/requests.md index d63fa586b..4a9e6fb8c 100644 --- a/docs/emitters/refine/requests.md +++ b/docs/emitters/refine/requests.md @@ -13,8 +13,10 @@ options](../contract/options.md)). The three first targets share one default: pa positional, in the order the primary binding's URI template names them (`HTTPBinding.URITemplate`), never in the order `Operation.Params` holds them, which follows the source's declarations rather than the URL; and every other input — query, header and cookie parameters and the body — is a field -of one request type for the operation, which it takes when it has any such input. Each target -declares that type its own way ([Go](../targets/go/operations.md#the-request-type), +of one request type for the operation, which it takes when it has any such input. A parameter a +client setting fills ([client settings](../operations/client.md#client-settings)) is an optional +field of the request type, whatever its location, since a call may leave it to the client. Each +target declares that type its own way ([Go](../targets/go/operations.md#the-request-type), [Kotlin](../targets/kotlin/operations.md#the-request-type)). Two properties decide the rule: 1. **Only the wire may fix an order.** A path template orders its parameters and a parameter list diff --git a/docs/emitters/refine/steps.md b/docs/emitters/refine/steps.md index de5824618..4369a1fc6 100644 --- a/docs/emitters/refine/steps.md +++ b/docs/emitters/refine/steps.md @@ -8,19 +8,20 @@ | `LowerLiterals` | `Literal{Value}` | typed constant / single-value type (incl. `symbol`) | [containers](../../ir/containers.md) | | `LowerModels` | `TypePlan.Flattened`, `Required`, `Undeclared`, `Unflattened`, `Base`/`Implements`/`Mixins`, `AdditionalProps`, `Additional`, `Positional` | a declaration over the flattened properties, an `Undeclared` member a required field typed as the catch-all types it, the catch-all holding the undeclared members, unknown members rejected on `closed`, an `Unflattened` conjunct lowered or degraded with a code | [models](../../ir/models.md) | | `ProjectVisibilityShapes` | `plan.ModelShape` per lifecycle | one declaration per shape, such as `User` / `UserCreate` / `UserPatch` | [properties](../../ir/properties.md#visibility) | -| `LowerOptionality` | `TypePlan.Required` (a member's wire presence, [TypePlan](../plan/types.md)) ⊥ `TypeRef.Nullable` ⊥ `Property.Presence`; `Facts.Component` | the four+ states in the target's forms; a reference on a cycle the target holds by value broken as its type system requires ([recursive references](models.md#recursive-references)) | [properties](../../ir/properties.md); INV8 | +| `LowerOptionality` | `TypePlan.Required` (a member's wire presence, [TypePlan](../plan/types.md)) ⊥ `TypeRef.Nullable`, a reference to a union with a variant admitting `null` counting as nullable ([unions](unions.md#a-null-a-variant-admits)) ⊥ `Property.Presence`; `Facts.Component` | the four+ states in the target's forms; a reference on a cycle the target holds by value broken as its type system requires ([recursive references](models.md#recursive-references)) | [properties](../../ir/properties.md); INV8 | | `LowerConstraints` | `Constraints` on properties/scalars/lists/maps | field-level validation metadata + a refine-built validation method structure | [constraints](../../ir/constraints.md) | -| `LowerUnions` | `Union{Variants, Exclusive, WireTagged, Discriminator}`, a polymorphic base's `Discriminator` or `Abstract` with `Facts.Subtypes`, `TypePlan.Union`, `TypePlan.Open` | the target's closed sum, one member per variant or subtype + a fallback on an open one | [unions](../../ir/unions.md); INV2 | +| `LowerUnions` | `Union{Variants, Exclusive, WireTagged, Discriminator}`, a polymorphic base's `Discriminator` or `Abstract` with `Facts.Subtypes`, `TypePlan.Union`, `TypePlan.Open`, `TypePlan.Inclusive` | the target's closed sum, one member per variant or subtype + a fallback on an open one; an inclusive union one optional member per variant, each set when its variant admits the value ([unions](unions.md#a-value-several-variants-admit)) | [unions](../../ir/unions.md); INV2 | | `LowerDiscriminator` | `Discriminator{Property/PropertyName/Index, Mapping→TypeID, Default, Envelope}` | a tag-dispatched decoder + typed factories | [models](../../ir/models.md)–[unions](../../ir/unions.md) | -| `LowerOpenEnums` | `Enum.Closed`, `ValueType`, `Members`, `Flags`, `FallbackMember` | the target's closed, open and flags forms; an open one keeps an unknown value | [enums](../../ir/enums.md) | +| `LowerOpenEnums` | `Enum.Closed`, `TypePlan.Open`, `ValueType`, `Members`, `Flags`, `FallbackMember` | the target's closed, open and flags forms; an open one keeps an unknown value | [enums](../../ir/enums.md) | | `ExtractInterfaces` | `Implements`, `OperationGroup` | interfaces / client interfaces for testability | [models](../../ir/models.md) | | `MapExternalTypes` | `External{Identity, Package, MinVersion}`, `Ctx.Types` | the type table's row for the identity ([type table](../contract/type-table.md)): imported library type + dependency-manifest entry; no row → opaque strategy + diagnostic | [containers](../../ir/containers.md) | | `MaterializeValues` | `Value{Kind, Num BigVal, Ref, Ctor}` | literals; `Num` → bignum/decimal, never float64; `Ctor` → call; `symbol` → string explicitly; a default documented, never written onto a request ([values](values.md)) | [values](../../ir/values.md) | -| `BuildRequestShaping` | `plan.BindingView`, `RequestContent`, the target's parameter shape ([request shaping](requests.md)) | request builders, per-protocol param binding, each operation's request type | [bindings](../../ir/bindings.md) | +| `BuildRequestShaping` | `plan.BindingView`, `RequestContent`, the target's parameter shape ([request shaping](requests.md)), the service's `ShapeHint.ClientParams` ([client settings](../operations/client.md#client-settings)) | request builders, per-protocol param binding, each operation's request type | [bindings](../../ir/bindings.md) | | `LowerHeaders` | `ResponsePlan.Headers` | a type with a field per declared header, each optional, read from the raw headers by wire name; a `WithResponse` method on a success ([return shapes](../operations/returns.md)), the failure value's and an `OtherResponse`'s `Headers` ([failures](../operations/errors.md), [return shapes](../operations/returns.md)) | [responses](../../ir/responses.md) | | `LowerPagination` | `plan.PagePlan` (item unwrap, cursor paths) | an auto-pager; pacing left to policy | [interaction-patterns](../../ir/interaction-patterns.md) | | `LowerStreaming` | `plan.StreamPlan` (direction; each side's content, framing and element) | a method per shape the operation offers; a stream over an element of any kind; a `bytes` side lowered as a file; terminal ends | [interaction-patterns](../../ir/interaction-patterns.md) | | `LowerLRO` | `plan.LROPlan` (final-state-via, poll/final ops, result path) | poller type + one poll and a poll until done, which returns the final type | [interaction-patterns](../../ir/interaction-patterns.md) | +| `UnwrapEnvelopes` | `ResponsePlan.Envelope` | in the response decoder, the flag read before the body, a failing body the runtime's failure value decoded as `Failure`, and the member at `Value` returned in the body's place; the envelope whole in a `WithResponse` method's `Response` ([envelopes](../plan/responses.md#envelopes), [return shapes](../operations/returns.md)) | [responses](../../ir/responses.md) | | `LowerErrors` | `plan.PlannedError` | no error type: each entry's conditions, body decoder and declared facts in the operation's response decoder, which fills the runtime's failure value ([failures](../operations/errors.md)) | [responses](../../ir/responses.md) | | `LowerOtherResponses` | `OpPlan.Others` | each entry's conditions in the operation's response decoder: `AsReturn` and a 2xx no entry answers read as `Return`, no body a nullable return or an ended pager or stream, any other a body decoder filling the runtime's `OtherResponse`, as a body that does not decode does ([return shapes](../operations/returns.md)) | [responses](../../ir/responses.md) | | `PlaceDeclarations` | `Provenance.Source`, `TypePlan.Reach`, the target's placement options | the output unit each declaration lands in: one package unless a partition or a group split moves it ([output units](output-units.md)) | [provenance](../../ir/provenance.md) | diff --git a/docs/emitters/refine/unions.md b/docs/emitters/refine/unions.md index ff468cd09..7f920a56d 100644 --- a/docs/emitters/refine/unions.md +++ b/docs/emitters/refine/unions.md @@ -33,8 +33,8 @@ survive for (de)serialization. **Unions never degrade to optional-field merges** *body* union into method arguments is an opt-in `ShapeHint` ([shaping hints](../contract/shaping-hints.md)), never a silent loss (the oagen counterexample, prior-art §1). A future TS/Python target lowers the *same* facts to a native `A | B | C`; the decision differs, the -fact base is identical. What a union holds when a value satisfies several variants of an `anyOf` is -#959's. +fact base is identical. What a union holds when several variants of an `anyOf` admit one value is +[below](#a-value-several-variants-admit). Within the untagged case, choosing *how* a variant is recognized on decode is refine's job, and ogen's `schema_gen_sum.go` is the battle-tested enumeration to lift: try, in order, explicit @@ -47,6 +47,64 @@ below, which keeps its bytes as they arrived — a degrade, never a whole-docume ([T-7](../testing/mandatory-tests.md)). That graceful-degradation margin is exactly what the ABI seam buys and the fused-pipeline generator cannot have. +A `oneOf` admits exactly one variant, so a value two of its variants admit breaks the document. What +it decodes to is `plan.Policy.ExclusiveMatch`'s ([plan policy](../plan/policy.md)): under `first`, +the default, the variant recognition finds first, as a lenient reader of a document whose variants +overlap needs; under `only`, a coded decode error, found by trying every variant whose kinds hold +the value, as an inclusive union does below. A `oneOf` whose variants the plan proves disjoint +([TypePlan](../plan/types.md#variants-that-can-share-a-value)) never meets the choice. + +## A null a variant admits + +A variant admits `null` when its reference is `Nullable` or its type is the `null` literal. `null` +holds nothing a variant could keep apart, so the union holds it itself: every reference to a union +with such a variant lowers as a nullable reference does +([optionality](models.md#optionality--nullability--presence-inv8)), a `null` never reaches a +variant, and a variant admitting nothing else lowers to no member. Both spellings the compiler can +give a null member (#618) then lower alike. The golden spec's single-result envelope declares +`result` as `anyOf: [{type: object, nullable: true}, {type: string, nullable: true}]`: an object, a +string or `null`, the last held by the reference. + +## A value several variants admit + +An `anyOf` lets one value satisfy several variants, and what the union holds then turns on +`TypePlan.Inclusive`, which the plan proves once +([TypePlan](../plan/types.md#variants-that-can-share-a-value)): + +- **Not inclusive.** No value is admitted by two variants, so holding the one that admits it loses + nothing, and the union lowers as a `oneOf` does: the target's closed sum, recognized as above. +- **Inclusive.** The union holds every variant that admits the value, each whole and under its own + name: an optional member per variant, set when that variant admits the value. Holding only the + first would misname the value and drop what the others declare. An Access identity provider would + decode as whichever provider is listed first, whatever its `type`, and its own `config` members + would be lost. + +A tag picks one variant, so a discriminated or key-tagged union is never inclusive, whatever +`Exclusive` says. + +A variant admits a value when the value decodes into it, with each codec, closed model, closed enum +and literal refusing what it refuses, and every model the value reaches carries each member its +`TypePlan.Required` lists. `Constraints` take no part, since they are validation's on every decoded +value. So whether a variant admits a value never depends on a bound. + +Decoding an inclusive union tries each variant whose kinds include the value's JSON kind, in +declaration order, and sets every one that admits the value. If none admits it, the order +[below](#a-value-no-variant-recognizes) applies: the fallback on an open union, a coded decode error +on a closed one. Trying every variant is what `anyOf` asks: no step of the ladder above can tell +whether a second variant admits a value too. + +Encoding writes the merge of what each set variant encodes. Objects merge member by member, in the +order the first set variant writes them, then each later variant's new members; arrays merge element +by element and must have one length; scalars must be equal. A member or element two variants write +must be the same JSON value, numbers compared by value, and a disagreement is an encode error, never +a silent pick. A union with no variant set refuses to encode and fails validation, and the fallback +is never set beside a variant. + +This is not the optional-field merge [the rule above](#variant-identity-and-recognition) forbids. +That merge pools the variants' properties into one bag, losing which variant a member belongs to and +admitting values no variant admits. Here each variant keeps its identity and its whole shape, and +validation still requires one. + ## A value no variant recognizes All four wire shapes decode in one order: the variant the tag or the key names, or the one diff --git a/docs/emitters/surface-verification.md b/docs/emitters/surface-verification.md index 1cd0c56ee..24b27bd47 100644 --- a/docs/emitters/surface-verification.md +++ b/docs/emitters/surface-verification.md @@ -4,11 +4,13 @@ Post-milestone. `EmitterOutput.Surface` is the neutral API-surface projection enums, methods, fields as neutral records — built by the **same walk** over `Origin`-tagged AST nodes that emit performs. -- **Neutral projection with IR IDs.** Both the freshly-generated SDK and an extracted - existing SDK (via a per-language extractor) project into one `verify.Surface`; each record carries - its **IR stable ID** (threaded from `Origin`), so cross-language findings correlate without - name-matching, and rename detection is an ID lookup rather than oagen's Jaccard-similarity - heuristics. `Generate` returns the generated side directly; only the extractor parses source. +- **Neutral projection with IR IDs.** Both the freshly-generated SDK and an extracted existing SDK + (via a per-language extractor) project into one `verify.Surface`; each record carries its **IR + stable ID** (threaded from `Origin`), so cross-language findings correlate without name-matching, + and a rename that keeps the entity's pointer is an ID lookup rather than oagen's + Jaccard-similarity heuristics. One that moves the pointer, such as a path renamed, is found by the + correlation key (#679) the manifest records. `Generate` returns the generated side directly; only + the extractor parses source. - **Injectable per-language severity.** The differ emits **neutral change records**; a per-target `verify.Severity` function decides breaking/additive (parameter names are public API in PHP, arity in Go, ~nothing in JS) — heuristics-as-policy (INV6). @@ -19,3 +21,57 @@ that emit performs. tracked in a distinct channel from a signature change — detectable without re-parsing because the plan already carries defaults as typed `Value`s. Every finding carries drift provenance (which stage caused it). + +## What a change record carries + +Some classifications turn on facts beyond the symbol that changed, so a neutral record carries them, +and a target's severity reads them rather than guessing: + +- **Direction.** Each record names the usages of the entity it concerns, from `Facts.Usage` + ([derived facts](../ir/derived-facts.md#usage)): an enum or a union gaining a member is additive + for a value the SDK sends and changes what a deployed SDK reading a response meets, unless the + type is `Open` ([plan policy](plan/policy.md)). A type used both ways is judged both ways, and the + more severe verdict counts. +- **Optionality in a response.** A response member going from required to optional breaks a typed + target, Go's `string` becoming `*string`, and is easy to miss, since the same change in a request + is additive. The record says which side the member is read on. +- **Bindings.** A changed method, path, parameter location or serialization style breaks a call + without changing a type, so it is a record of the behavioral channel, beside a changed default. + +Two channels answer two questions. The structural channel asks whether a caller's code still +builds against the regenerated SDK. The behavioral channel asks whether a deployed SDK, built +against the old document, still works against the API the new one describes, which a new member of +a closed response enum does not. + +## Severity, seeded + +Each target's `verify.Severity` starts from the rules every typed target shares, one verdict per +channel, and states where it departs ([Go](targets/go/overview.md#merging-and-surface-extraction), +[Kotlin](targets/kotlin/overview.md#merging-and-surface-extraction)): + +| Change | Structural: callers' code | Behavioral: deployed SDKs | +|---|---|---| +| An entity removed, or renamed with no correlation key carrying it | breaking | breaking | +| A request member or parameter added | additive if optional, breaking if required | breaking if required | +| A request member made required | breaking | breaking | +| A response member added | additive | additive | +| A response member made optional | breaking on a typed target | breaking | +| A union or enum gaining a member, read in a response | the target's: breaking where a match over it must be exhaustive, as a Kotlin `when` over an `enum class` | breaking unless `Open` | +| A union or enum gaining a member, only sent | additive | additive | +| A success status added | breaking where it changes a return ([return shapes](operations/returns.md)) | breaking | +| A failure status added | additive | additive | +| A media type removed | breaking | breaking | +| A binding changed: method, path, parameter location or style | none | breaking | +| A default changed | none | reported | + +## The version a regeneration takes + +A regeneration's version bump follows from what changed, never from a digest of the spec's text. +The structural channel sets it, since an SDK's version is a promise to its callers' code: a +breaking record makes it major, an additive one minor. A behavioral record makes it at least minor, +since the regenerated SDK behaves differently from the last. A regeneration with none of those that +still changed an artifact, through a step whose version moved +([verification](verification.md#during-steps-carry-versions)) or a document whose wording did, +takes a patch, and one that changed no artifact takes none, which the manifest's digests show +([write / integrate](write-integrate.md#what-the-manifest-records)). + diff --git a/docs/emitters/targets/go/emit.md b/docs/emitters/targets/go/emit.md index d56490d14..b66f76904 100644 --- a/docs/emitters/targets/go/emit.md +++ b/docs/emitters/targets/go/emit.md @@ -41,11 +41,13 @@ type Iface struct{} // any ## Why a domain AST → text → `go/format`, not `go/ast` directly `go/ast` is awkward to *synthesize* (it wants a `token.FileSet` and real positions and cannot carry -generator intent like a field's `Origin` IR ID or a `Docs.Refs` span). The domain AST is -smaller, holds provenance and wire metadata on its nodes, and prints to text that `go/format` then -makes canonical. `go/ast` is used in exactly one place — re-parsing *existing* files for additive -merge ([write / integrate](../../write-integrate.md)) — where its parser is the right tool. The two -Go trees are kept aligned by ID-keying: "a symbol I generated" ↔ "a symbol I parsed" is a lookup. +generator intent like a field's `Origin` IR ID or a `Docs.Refs` span). The domain AST is smaller, +holds provenance and wire metadata on its nodes, and prints to text that `go/format` then makes +canonical. A syntax tree of Go is used in exactly one place — re-parsing *existing* files for +additive merge ([write / integrate](../../write-integrate.md)) — and that one is `dst`'s, which +keeps comments on their nodes where `go/ast` cannot +([merging](overview.md#merging-and-surface-extraction)). The two Go trees are kept aligned by +ID-keying: "a symbol I generated" ↔ "a symbol I parsed" is a lookup. ## Runtime templates diff --git a/docs/emitters/targets/go/lowering.md b/docs/emitters/targets/go/lowering.md index ffa288524..e77430559 100644 --- a/docs/emitters/targets/go/lowering.md +++ b/docs/emitters/targets/go/lowering.md @@ -17,8 +17,8 @@ Go's normative lowering of each construct of the [lowering table](../../refine/l | anonymous decorator: a `Model` with only a `Base`, or a `Scalar` with neither `Constraints` nor `Encoding`, beside annotations ([decorators](../../refine/decorators.md)) | no declaration: each reference lowered as one to the first type along its `Base` that is not a decorator, its docs, deprecation and examples folded onto the carrier | | `Model` whose `Base` is no model, not a decorator | with no property, catch-all or discriminator of its own, lowered as the `Scalar` row lowers an alias over that base, its constraints, docs and examples binding the position ([scalars](../../../ir/scalars.md)); a union base beside properties distributed over the variants as [models](../../../ir/models.md#composition-beside-a-union) distributes a union co-declared with a composition; anything else a diagnosed degrade | | `AdditionalProps` / `Additional` | catch-all `map` as the embedded fallback; `closed`→`RejectUnknownMembers(true)`, pinned per model ([`User`](models.md#user-in-go)) | -| `Union` (four wire shapes, [unions](../../refine/unions.md#the-wire-shapes)) | a struct holding a sealed interface + marker + a wrapper per variant + a tag-dispatch (de)serializer on the struct; `Open` → an `Unknown` fallback | -| `Enum` open/closed/flags | consts named by enum then member (`StatusActive`, [enums](types.md#enums)): closed→typed consts; open→string+consts+`Unknown`; flags→bitset; `FallbackMember` on decode | +| `Union` (four wire shapes, [unions](../../refine/unions.md#the-wire-shapes)) | a struct holding a sealed interface + marker + a wrapper per variant + a tag-dispatch (de)serializer on the struct; `Open` → an `Unknown` fallback; an inclusive `anyOf` → a struct with a pointer per variant ([unions](unions.md#an-inclusive-union)) | +| `Enum` open/closed/flags | a named type over the value type's row, consts named by enum then member (`StatusActive`, [enums](types.md#enums)): closed→typed consts, strict decode; open, or `TypePlan.Open`→consts+`Unknown`; flags→bitset; `FallbackMember` on decode; a value type with no Go constant → a named `string` over the wire text, its codec writing the value type's JSON kind, + a `Value()` of the row's type | | `List`/`MapT`/`Tuple` | `[]T`/`map[K]V`/positional struct (optional past `MinItems`, `Rest`→trailing slice); `List.Encoding`→serializer mode | | `Literal` | typed constant (incl. `symbol`) | | `External` | the type-table row for its `Identity` ([type table](../../contract/type-table.md)): imported library type + dep-manifest; no row → opaque handle, passthrough + diagnostic | @@ -51,9 +51,9 @@ lowers to in Go. | `ProjectVisibilityShapes` | `User` / `UserCreate` / `UserPatch` structs | | `LowerOptionality` | the four+ states → `T` / `*T` / `Opt[T]`/`Nil[T]`/`OptNil[T]` / `omitzero` + presence bit; a reference on a cycle the target holds by value held through a pointer ([recursive references](models.md#recursive-references)) | | `LowerConstraints` | field-level `validate` tags + a refine-built `Validate()` method structure | -| `LowerUnions` | a struct holding a sealed interface + marker method + a wrapper per variant or subtype + a fallback on an open one | +| `LowerUnions` | a struct holding a sealed interface + marker method + a wrapper per variant or subtype + a fallback on an open one; on an inclusive union, a struct with a pointer per variant | | `LowerDiscriminator` | tag-dispatched `UnmarshalJSONFrom` on the union's struct + typed factories | -| `LowerOpenEnums` | closed → typed consts; open → `string`+consts+`Unknown(raw)`; flags → bitset | +| `LowerOpenEnums` | closed → typed consts; open → consts+`Unknown(raw)`; flags → bitset; each over the value type's row, or the wire text where it has no constant | | `ExtractInterfaces` | Go interfaces / client interfaces for testability | | `LowerPagination` | auto-pager yielding `iter.Seq2[T, error]`; pacing left to policy | | `LowerStreaming` | a method per shape the operation offers; `Stream[T]` with `Recv()`/`Send()` over an element of any kind; a `bytes` side lowered as a file; terminal ends | diff --git a/docs/emitters/targets/go/operations.md b/docs/emitters/targets/go/operations.md index a2e0d6916..11db844ad 100644 --- a/docs/emitters/targets/go/operations.md +++ b/docs/emitters/targets/go/operations.md @@ -29,8 +29,9 @@ operation has no input ([request shaping](../../refine/requests.md)). `Stream[T] a caller that stops early must do. What `T` is follows from `Recv`: - an `Element` is what its type lowers to. A union's is its struct - ([`Payment`](unions.md#payment-in-go)), and `Recv` returns `io.EOF` after a variant whose - `Variant.Event.Terminal` is true, so an event union needs nothing of its own beyond that. + ([`Payment`](unions.md#payment-in-go)), and `Recv` returns `io.EOF` after an element holding a + variant whose `Variant.Event.Terminal` is true, so an event union needs nothing of its own beyond + that. - no `Element` is `Stream[string]`, each frame's text, unread. - a `bytes` side has no frames and no `Stream`: `Recv` lowers as a file does, to `*RawBody` ([return shapes](returns.md)), and `Send` to an `io.Reader` field of the request type. @@ -60,4 +61,6 @@ operations](../../operations/long-running.md). ## The client The client's settings ([the client](../../operations/client.md)) are functional options — -`NewClient(WithRegion(RegionEU))`, `WithServer(…)`, `WithBaseURL(u)`. +`NewClient(WithRegion(RegionEU))`, `WithServer(…)`, `WithBaseURL(u)`. A [client +setting](../../operations/client.md#client-settings) is one more, `WithAccountID("…")`, and the +request type's field for it a pointer, `AccountID *string`, nil taking the client's. diff --git a/docs/emitters/targets/go/overview.md b/docs/emitters/targets/go/overview.md index fb925593a..84808e070 100644 --- a/docs/emitters/targets/go/overview.md +++ b/docs/emitters/targets/go/overview.md @@ -39,8 +39,42 @@ check](../../refine/output-units.md#the-size-check) reports and its split by gro ## Merging and surface extraction -The Go [merge adapter](../../write-integrate.md) (`emitters/golang/mergeadapter.go`) uses `go/ast` -to parse existing files and append only new top-level symbols, never delete. +The Go [merge adapter](../../write-integrate.md) (`emitters/golang/mergeadapter.go`) parses an +existing file with `github.com/dave/dst`, a decorated syntax tree, and appends only new top-level +symbols, never deleting one. `dst` keeps each comment on the node it describes, so a declaration +moved from the new generation's file into the old one carries its doc comment and a +`@morphic:keep` marker stays where its region is; `go/ast` keeps comments in a list of their own by +byte offset, so a node moved between two files leaves them behind. + +A spike settled it, at Go 1.27.3 with `dst` v0.28.0. A generated file holding a keep region gained +a type and a method from the next generation, three ways: + +- **`go/ast` nodes moved, printed by `go/printer`:** the moved declarations lost their doc + comments, which landed above `GetZone`, and one of them inside `GetZone`'s body. +- **`go/ast` to find the declarations, their source text spliced in, `astutil.AddImport`:** correct, + the region untouched and the import added. It reaches top-level declarations alone, so adding a + field to a struct, which a merge at property level would need ([open + questions](../../open-questions.md), 5), would be a splice inside one with no tree to check it. +- **`dst`:** correct, imports managed by its restorer, and a file using generics, a range-over-func + iterator and `min` round-trips byte for byte. + +`dst` is MIT-licensed, at v0 and maintained in bursts, its last release in 2026. So its version is +pinned exactly, `emitters/golang` alone may import it ([package layout](../../packages.md)), and an +upgrade reruns the spike's cases as tests of the adapter. Go's extractor ([surface verification](../../surface-verification.md)) reads the existing package's -exported API, as `apidiff` does. +exported API with `go/types`, and each record it builds carries the IR ID its declaration's +`Origin` names, which no Go tool records. + +`golang.org/x/exp/apidiff` compares two Go packages and labels each change compatible or not, which +is the Go half of the structural channel's question. It is the test oracle for Go's severity, not +the differ and not the severity function: over the surface corpus, a differential test holds each +structural verdict Go's `verify.Severity` gives to apidiff's on the same two generated packages, and +a disagreement is either a severity bug or a divergence the severity's documentation states. It +cannot be the differ, since its records carry no IR ID, nor the severity, since it answers only +whether a caller's Go code builds: given an enum gaining a constant it says compatible, which it is +for a caller's code, while the behavioral channel still reports what a deployed SDK reading a closed +response enum meets ([surface +verification](../../surface-verification.md#what-a-change-record-carries)). As a test-only +dependency it needs no compatibility promise from `x/exp`, which gives none; it was under active +development, BSD-3-Clause, at the October 2026 revision measured. diff --git a/docs/emitters/targets/go/returns.md b/docs/emitters/targets/go/returns.md index eae2b1909..50265c0b8 100644 --- a/docs/emitters/targets/go/returns.md +++ b/docs/emitters/targets/go/returns.md @@ -12,6 +12,9 @@ How each [return shape](../../operations/returns.md) lowers in Go. `func (c *Client) GetZoneWithResponse(ctx) (*Response[*Zone, GetZoneHeaders], error)`. The `Response`'s `Body` is `struct{}` for `void`, and its `Headers` is `struct{}` where they declare none. +- **An envelope.** `GetZone` returns `*Zone`, and `GetZoneWithResponse`'s `Response` holds the + envelope in `Envelope any`, a `*GetZoneResponse` there and nil on an operation with no envelope, + as `OtherResponse.Body` holds a body of any kind. - **Another declared response.** With no body, `DeleteMethod` returns a nil `*Deleted` and a nil error, and a stream's `Recv` returns `io.EOF`. diff --git a/docs/emitters/targets/go/types.md b/docs/emitters/targets/go/types.md index cb1274437..752d6bdee 100644 --- a/docs/emitters/targets/go/types.md +++ b/docs/emitters/targets/go/types.md @@ -30,10 +30,33 @@ for one. ## Enums -Go cannot express open enums, so `Closed=false` lowers to a named string type + typed consts + a -passthrough that preserves unknown values and honors `FallbackMember` on decode; `Closed=true` -lowers to typed consts with strict decode; `Flags=true` → bitset. A `ValueType` whose Go type has -no constant form, such as `date`, `uuid` or `decimal`, has no Go lowering yet (#1202). +An enum is a named type over its `ValueType`'s Go row, with a typed constant per member. Go cannot +express open enums, so `Closed=false` lowers to that type + its consts + a passthrough that +preserves unknown values and honors `FallbackMember` on decode, as does an enum `TypePlan.Open` +marks ([enums](../../refine/enums.md)); `Closed=true` lowers to that type + its consts with strict +decode; `Flags=true` → bitset. + +Go declares constants of boolean, numeric and string types alone, so a `ValueType` whose row is none +of them — `date`, `time`, `datetime`, `datetime_offset`, `duration`, `uuid`, `bytes`, `decimal`, and +`integer` or `number` where a row maps them exactly — lowers over the member's wire form ([a member +on the wire](../../refine/enums.md#a-member-on-the-wire)): a named type over `string`, each constant +the member's wire text, whose codec writes that text as the value type's JSON kind, a number for a +`decimal`, and decodes by comparing as JSON does, so the constant of a decimal `0.1` is the value a +`0.10` decodes to. A method returns the value as the row's type, decoding the text through the row's +codec, as [the type table](../../contract/type-table.md) builds any literal of a mapped type: + +```go +type Day string // date, enum: ["2020-01-01"] + +const Day20200101 Day = "2020-01-01" + +func (d Day) Value() (sdk.Date, error) // the member as the row's type +``` + +Two designs lost to it. Package-level variables built through the codec cover every type, but a +`var` can be assigned, and `==` is wrong for `time.Time` and absent for a decimal over `math/big`, +so a strict decoder and a caller's `switch` would each need a comparison of their own. A named type +over the row's type has no constant to give a caller at all. A Go constant is declared at package level, where every enum's constants meet, so Go names a member's constant by its enum's identifier, as rendered, followed by the member's name diff --git a/docs/emitters/targets/go/unions.md b/docs/emitters/targets/go/unions.md index 363a31486..226fc2ad5 100644 --- a/docs/emitters/targets/go/unions.md +++ b/docs/emitters/targets/go/unions.md @@ -43,6 +43,30 @@ The fallback ([the shared rule](../../refine/unions.md#a-value-no-variant-recogn variant, `PaymentUnknown`, holding the value as `jsontext.Value`. It re-encodes as the same JSON value, though json/v2 normalizes whitespace and string escapes as it writes. +## An inclusive union + +An inclusive union ([the shared rule](../../refine/unions.md#a-value-several-variants-admit)) holds +every variant a value admits, so it is a struct with a pointer per variant and no interface: + +```go +type Target struct { // anyOf: [Zone, Account], which one value can fit + Zone *Zone // set when Zone admits the value + Account *Account // set when Account admits it, beside Zone or alone + Unknown jsontext.Value // an open union's fallback, never set beside a variant +} +``` + +- **A pointer per variant**, whatever the variant's Go type, so nil means "does not admit it" for a + slice or a string too. Each field is named as the variant's wrapper in a closed sum is, so two + variants that lower to one Go type stay apart. +- **The struct declares the codec pair**, as a closed sum's struct does. `UnmarshalJSONFrom` buffers + the value (`dec.ReadValue()`), decodes it into each variant under `dec.Options()`, and keeps each + that admits it: the decode succeeds, and the unexported presence check generated beside the + variant's `Validate`, which reads no `Constraints`, finds every required member. `MarshalJSONTo` + writes the merge. +- **`Validate` reports a struct with no field set**, and validates each field that is. With no sum + to close, it has no marker, no wrapper and no switch-completeness test. + ## Discriminators `LowerDiscriminator` emits a tag-dispatched `UnmarshalJSONFrom` that peeks the discriminator diff --git a/docs/emitters/targets/kotlin/lowering.md b/docs/emitters/targets/kotlin/lowering.md index 83b8be2f7..328addc9b 100644 --- a/docs/emitters/targets/kotlin/lowering.md +++ b/docs/emitters/targets/kotlin/lowering.md @@ -19,8 +19,8 @@ where Multiplatform differs from the JVM ([Multiplatform common code](multiplatf | anonymous decorator: a `Model` with only a `Base`, or a `Scalar` with neither `Constraints` nor `Encoding`, beside annotations ([decorators](../../refine/decorators.md)) | no declaration: each reference lowered as one to the first type along its `Base` that is not a decorator, its docs, deprecation and examples folded onto the carrier | | `Model` whose `Base` is no model, not a decorator | with no property, catch-all or discriminator of its own, lowered as the `Scalar` row lowers an alias over that base, its constraints, docs and examples binding the position ([scalars](../../../ir/scalars.md)); a union base beside properties distributed over the variants as [models](../../../ir/models.md#composition-beside-a-union) distributes a union co-declared with a composition; anything else a diagnosed degrade | | `AdditionalProps` / `Additional` | catch-all `Map` filled by a serializer that reads the whole object ([unknown members](models.md#unknown-members)); strict `Json`, `@JsonIgnoreUnknownKeys` on each other open model | -| `Union` (four wire shapes, [unions](../../refine/unions.md#the-wire-shapes)) | sealed interface + one wrapper class per variant + a `KSerializer` per wire shape; `Open` → an `Unknown` fallback; KMP: not exhaustive in Swift | -| `Enum` open/closed/flags | closed → `enum class` with `@SerialName` members, `FallbackMember` on decode; open → value class over the value type with a constant per member, which keeps an unknown value; flags → value class over an integer | +| `Union` (four wire shapes, [unions](../../refine/unions.md#the-wire-shapes)) | sealed interface + one wrapper class per variant + a `KSerializer` per wire shape; `Open` → an `Unknown` fallback; KMP: not exhaustive in Swift; an inclusive `anyOf` → a class with a nullable property per variant ([unions](unions.md#an-inclusive-union)) | +| `Enum` open/closed/flags | closed → `enum class` with `@SerialName` members on a `string` enum, a generated `KSerializer` writing each member's JSON kind on any other ([enums](types.md#enums)), `FallbackMember` on decode; open, or `TypePlan.Open` → value class over the value type with a constant per member, which keeps an unknown value; flags → value class over an integer | | `List`/`MapT`/`Tuple` | `List`/`Map`/a class with a positional serializer (nullable past `MinItems`, `Rest` → trailing `List`) | | `Literal` | `const val`, or an `object` for a non-primitive value | | `External` | the type-table row for its `Identity`, each platform's own; no row → diagnostic | diff --git a/docs/emitters/targets/kotlin/naming.md b/docs/emitters/targets/kotlin/naming.md index 3f3311fc3..aae66cbac 100644 --- a/docs/emitters/targets/kotlin/naming.md +++ b/docs/emitters/targets/kotlin/naming.md @@ -4,9 +4,30 @@ What each [scope](../../refine/reserved-names.md) reserves in Kotlin: +- **The units**: `build`, `buildsrc` in any letter case, `gradle` and `src`, the names Gradle gives + a meaning at a build's root, where a group's module sits ([output + units](overview.md#output-units)). `build` is the root project's output, so a `clean` of the root + deletes a module there, sources and all. `buildSrc` is Gradle's build-logic build, found without + regard to case on a case-insensitive file system such as macOS's, so a module named `buildsrc` + fails the build there. `gradle` holds the wrapper and the version catalog, and `src` the root + project's sources, so a module named either would share their directory. +- **A unit**: `Client` and, in a group's unit, its sub-client; and in the root unit every runtime + name the target declares there, which each runtime template declares beside it, as a function's + names are ([reserved names](../../refine/reserved-names.md)): `ApiFailure`, `ApiException`, + `ClientException`, `ServerException`, the exception of each kind the error taxonomy names + (`NotFoundException`), `Fault`, `OtherResponse`, `OtherResponseException`, `Response`, `RawBody`, + `Optional`, `Duration`, `Authenticator` and `Transport`, and on Multiplatform `Decimal` and + `OffsetDateTime`. - **A type**: `equals`, `hashCode`, `toString`, `Builder` and `Companion`. -The sets for the units and for a unit are not stated yet (#1187). +Neither a hard keyword nor `kotlin` is a units name. A hard keyword as a package segment takes the +escape below, which kotlinc accepts in a `package` and an `import` alike: `fun`, `object`, `when`, +`is`, `in` and `typealias`, backticked, compile for the JVM, JS and macOS Native and link into a +framework, and those with `class`, `val`, `interface`, `as`, `true`, `null`, `this`, `super`, +`package`, `return`, `throw` and `try` on the JVM. kotlinc refuses `kotlin` only as a package's +first segment ("Only the Kotlin standard library is allowed to use the 'kotlin' package"), which is +the root's and so the options' to name. Each of these was measured with Kotlin 2.4.0 and Gradle +9.8.2 on darwin/arm64. ## The escape diff --git a/docs/emitters/targets/kotlin/operations.md b/docs/emitters/targets/kotlin/operations.md index dbe06129f..1dd3c9107 100644 --- a/docs/emitters/targets/kotlin/operations.md +++ b/docs/emitters/targets/kotlin/operations.md @@ -15,4 +15,5 @@ Kotlin's request type ([request shaping](../../refine/requests.md)) is a class b ## The client The client's settings ([the client](../../operations/client.md)) are setters on the client's -builder. +builder. A [client setting](../../operations/client.md#client-settings) is one more, +`accountId("…")`, and the request type's property for it is nullable, null taking the client's. diff --git a/docs/emitters/targets/kotlin/overview.md b/docs/emitters/targets/kotlin/overview.md index 27dd76408..4e035a4c0 100644 --- a/docs/emitters/targets/kotlin/overview.md +++ b/docs/emitters/targets/kotlin/overview.md @@ -35,7 +35,9 @@ Codecs are generated `KSerializer`s, the counterpart of Go's A Kotlin [unit](../../refine/output-units.md) is a package, and also a Gradle module of its own unless it is the root, since kotlinc compiles a module whole and a Multiplatform framework exports -what its modules declare ([scale](scale.md)). A group's unit is `.`. +what its modules declare ([scale](scale.md)). A group's unit is `.`, and its module is +the directory `` at the build's root, beside the root project's own `src`, so the names +Gradle gives a meaning there are reserved ([naming](naming.md#reserved-names)). Kotlin's limit counts a Gradle module's declarations, and its default is set when the Kotlin targets are first measured ([open questions](../../open-questions.md)). diff --git a/docs/emitters/targets/kotlin/returns.md b/docs/emitters/targets/kotlin/returns.md index b4b713c80..1bf6e129c 100644 --- a/docs/emitters/targets/kotlin/returns.md +++ b/docs/emitters/targets/kotlin/returns.md @@ -8,6 +8,8 @@ How each [return shape](../../operations/returns.md) lowers in Kotlin. - **No body.** `Unit`. - **Declared headers.** `suspend fun getZoneWithResponse(): Response`. The `Response`'s `Body` is `Unit` for `void`, and its `Headers` is `Unit` where they declare none. +- **An envelope.** `suspend fun getZone(): Zone`, and `getZoneWithResponse()`'s `Response` holds + the envelope in `val envelope: Any?`, null on an operation with no envelope. - **Another declared response.** With no body, `suspend fun deleteMethod(): Deleted?`. ## `OtherResponse` diff --git a/docs/emitters/targets/kotlin/types.md b/docs/emitters/targets/kotlin/types.md index 802d3f4ca..a17049c39 100644 --- a/docs/emitters/targets/kotlin/types.md +++ b/docs/emitters/targets/kotlin/types.md @@ -26,3 +26,11 @@ the root's type, and its serializer, in every signature. An enum's forms are in the [lowering table](lowering.md). A Kotlin entry lives in its enum class and takes the member's name alone (`ACTIVE`, as the naming policy cases it). + +`kotlinx.serialization` writes an `enum class` as a string, its entry's `@SerialName`, which is a +member's wire form only for a `string` enum ([a member on the +wire](../../refine/enums.md#a-member-on-the-wire)). Any other closed enum is an `enum class` whose +`KSerializer` is generated: it writes each entry's wire form as the value type's JSON kind, a +boolean for `bool`, a number for `integer`, `number` and `decimal`, and decodes by comparing as JSON +does. An entry of a value type a row maps, such as a `date`, holds the member as that row's type, +`val value: LocalDate`, built through the row's codec when the class loads. diff --git a/docs/emitters/targets/kotlin/unions.md b/docs/emitters/targets/kotlin/unions.md index 25c7a4277..c03216fa4 100644 --- a/docs/emitters/targets/kotlin/unions.md +++ b/docs/emitters/targets/kotlin/unions.md @@ -12,3 +12,12 @@ Kotlin's variant classes are the same wrapper as [Go's](../go/unions.md), for th The fallback ([the shared rule](../../refine/unions.md#a-value-no-variant-recognizes)) is `Payment.Unknown`, holding a `JsonElement`. + +## An inclusive union + +An inclusive union ([the shared rule](../../refine/unions.md#a-value-several-variants-admit)) is a +final class with a nullable property per variant, named as the variant's wrapper class is, and +`unknown: JsonElement?` on an open one. Its `KSerializer` reads the value as a `JsonElement`, +decodes it into each variant through `Json.decodeFromJsonElement`, and keeps each that admits it: +the decode succeeds and no required member was absent. It encodes the merge. `validate()` reports a +value with no property set, and no `init` check does, since a decoded value always has one. diff --git a/docs/emitters/testing/mandatory-tests.md b/docs/emitters/testing/mandatory-tests.md index 817580617..8864bb87b 100644 --- a/docs/emitters/testing/mandatory-tests.md +++ b/docs/emitters/testing/mandatory-tests.md @@ -21,8 +21,15 @@ throughout these documents. 7. **T-7 · Diagnostic-not-silence** — every un-lowerable construct emits a coded diagnostic, never a silent `any` and never a panic, a fact outside M3's scope included ([what M3 lowers](../milestones.md#what-m3-lowers)). -8. **T-8 · Manifest ID-keying** — the entity→symbol map survives path/name churn. -9. **T-9 · Policy precedence** — declared `Retryable`/`Fault`/`Idempotency` override policy defaults. +8. **T-8 · Manifest ID-keying** — the entity→symbol map survives a rename of what an ID is not + built from, such as an `operationId`, and, through the correlation key, a path or component + rename. +9. **T-9 · Policy precedence** — declared `Retryable`/`Fault`/`Idempotency` override policy defaults + at a document-wide setting and at an operation's, in every layer. A project's document-wide + setting beats a per-operation one the document embeds, an operation's own setting beats its + layer's document-wide one, an operation ID no operation has is reported, and an embedded value a + higher layer replaces draws an info naming both + ([layers](../contract/runtime-policy.md#layers-and-operations)). 10. **T-10 · Step-trace golden** — the ordered refine step list produces a pinned, deterministic trace (guards implicit inter-step coupling). 11. **T-11 · Architecture import-graph** — the layering in [package layout](../packages.md). @@ -63,8 +70,9 @@ throughout these documents. 19. **T-19 · Reserved scopes** — a fixture naming its properties, parameters, path parameters, enum members and union variants after every name each scope of [reserved names](../refine/reserved-names.md) reserves, after characters alone and after names that begin - with a digit, and its groups after the standard-library packages its files import, generates - code that compiles on every target, with every template's declared names held to what it binds. + with a digit, and its groups after the standard-library packages its files import and after + every units name, generates code that compiles on every target, a `clean` of a Kotlin build + leaving every module's sources, with every template's declared names held to what it binds. 20. **T-20 · Recursive types** — a model that holds itself directly, through a base, through a tuple and through another model, and one whose cycle passes through a list or a union, generate code that compiles on every target, and exactly the references on a cycle Go holds by value are held @@ -142,3 +150,110 @@ throughout these documents. open. An option beside it is taken in priority order, an option ANDing it with a declared scheme needs both, and the service's credentials are never sent in its place ([auth](../operations/auth.md#unresolved-schemes)). +29. **T-29 · Inclusive unions** — on every target, an `anyOf` over two open models that one value + fits decodes that value into both and re-encodes it as an equal JSON value; a member the two + write at different values is an encode error, and a union with neither set refuses to encode. + An `anyOf` the plan proves disjoint, by JSON kind, by enum values and by a member one model + requires and the other cannot hold, lowers as a closed sum and decodes each value into the one + variant admitting it. A `null` that two nullable variants admit, or a `null` literal variant + does, is held by the reference, as a nullable one's is. A value no variant admits takes the + fallback on an open union and is a coded error on a closed one + ([unions](../refine/unions.md#a-value-several-variants-admit), + [TypePlan](../plan/types.md#variants-that-can-share-a-value)). +30. **T-30 · Plan identity** — one document under one policy builds plans of one `Version` across + processes and runs, and the policy round-trips through JSON unchanged; changing any policy + field, the `Revision`, or the document outside `sources` changes the `Version`, and changing + only `sources` does not. A plan golden that changes at an unmoved `Revision` fails + ([the plan's version](../plan/overview.md#the-plans-version)). +31. **T-31 · Envelopes** — under an envelope matched by `Members` and one matched by `Bases`, on + every target, a method returns the member at `Value`, its `WithResponse` method holds the + envelope whole, and a 2xx whose flag holds `Fails` is a failure with no fault or kind whose body + decodes as `Failure`, even where the success type admits only `true`. A paginated operation's + page is read whole, its flag still read. A body naming only some of the members, an operation + `Except` lists and a failure body are read as they would be with no envelope, and the default + policy unwraps nothing ([envelopes](../plan/responses.md#envelopes), + [plan policy](../plan/policy.md#envelopes)). +32. **T-32 · Embedded policy and hints** — a mapping reading a key on the document, on an operation + and on a parameter configures the document, that operation and that parameter's hint; a key on + a node it cannot configure, a value of the wrong shape and a mapping naming a key of a reason + other than `vendor_extension` are each reported, the entry staying in `Unmodeled`; the default + mapping reads nothing, and `Disabled` reads nothing however the mapping is written + ([embedded policy](../contract/embedded-policy.md)). +33. **T-33 · Client settings** — on every target, a service hint naming a path parameter makes it + a client setting, which every operation declaring it in that wire form sends unless the call + gives its own; a call with neither is refused unsent; an operation declaring it at another + wire form keeps it as an input and is reported once; no parameter becomes a setting unless a + hint names it, and the plan reports each candidate as an info + ([client settings](../operations/client.md#client-settings)). +34. **T-34 · Effects** — a declared `Idempotency` of each kind beats the method, a `GET` with none + is safe and idempotent, a `PUT` and a `DELETE` idempotent, a `POST` neither, each derived one + marked `Inferred`; turning `MethodSemantics` off leaves every undeclared operation neither; and + on every target a request that got no response is sent again for an idempotent operation and + never for a `POST`, under the default `Retry.Connection` + ([effects](../plan/policy.md#an-operations-effects-come-from-its-method)). +35. **T-35 · Open enums and overlapping oneOfs** — on every target, a closed enum a response carries + decodes a value it does not list and re-encodes it as it came, its constants naming only the + listed members, while one only ever sent stays strict and turning `OpenEnums` off restores + strict decoding; a `oneOf` value two variants admit decodes into the first under `first` and is + a coded error under `only` ([plan policy](../plan/policy.md#an-enum-a-client-reads-is-open), + [unions](../refine/unions.md#variant-identity-and-recognition)). + +36. **T-36 · Enums over every value type** — on every target, an enum of each primitive a + `ValueType` can name, `bool`, `integer`, `number`, `decimal`, `date`, `uuid` and `bytes` among + them, closed and open, encodes each member as its value type's JSON kind and decodes it back to + the same constant, a number spelled with trailing zeros included; where the target's constants + cannot hold the type, the member's value is reachable as the row's type, and no member is ever + written as a string its value type does not admit + ([a member on the wire](../refine/enums.md#a-member-on-the-wire)). +37. **T-37 · Shipped names** — on every target, a generation whose operation collides with nothing, + then a revision adding one that collides with it, regenerated against the first generation's + manifest, keeps the first symbol unchanged and gives the newcomer a suffix; the first + generation is byte-identical in both declaration orders; a renamed `operationId` renames its + symbol; and an ID that now names another entity, its correlation key differing, is not pinned + ([collisions](../refine/naming.md#collisions-and-the-names-a-generation-shipped)). +38. **T-38 · Manifest and integration** — a regeneration whose inputs all match writes nothing; a + spec reformatted into the same IR is the same generation; a generated file edited outside its + regions is reported and not overwritten, while one checked out with other line endings is not + an edit; a region follows its entity into the file a rename moves it to, and one whose entity + is gone is reported and its file kept; a file written once is never written again; a hint, an + override and an overlay key matching nothing each draw `emitters/unmatched-key`, naming the + entity of the same correlation key; a README's text outside its blocks and its frozen blocks + survive byte for byte, and its table of contents lists the user's headings; and a manifest of + an older `Schema` is migrated, one of a newer refused + ([write / integrate](../write-integrate.md)). +39. **T-39 · Docs** — the docs emitter renders every response, error case and content an operation + declares, the primary first, and a type's request and response shapes apart, a write-only + property in no response table; every operation, page-worthy type and property has a route or + anchor, and two entities resolving to one URL is an error naming both, also when the URL + templates are changed; every link and `Docs.Refs` span resolves through the route plan; a + version switcher links one entity across snapshots and an entity a version lacks to that + version's page saying so; agent documents and `llms.txt` share the pages' URLs, escape spec + text and pass no raw HTML; and a sample comes from each SDK target, a missing one a diagnostic + ([docs and mocks](../docs-and-mock.md)). +40. **T-40 · One set of examples** — a declared example is used everywhere and labelled authored; + where none is declared, the value synthesized is valid against its constraints, within the + policy's depth and item bounds, marked `Inferred`, identical across regenerations and in both + declaration orders, and the same in the docs, the samples, an SDK's tests and the mock; one the + synthesizer cannot make valid is reported and left out + ([one set of examples](../docs-and-mock.md#one-set-of-examples)). +41. **T-41 · Unhonored rows** — a document exercising a row a target does not honor draws + `emitters//unhonored-row` naming the row and its locations before `Generate` runs; a row + the target honors draws nothing; and every key of the capability matrix has a detector in + `irfacts.Rows` or a stated reason for none ([verification](../verification.md)). +42. **T-42 · The verify stage** — every golden output of every target builds, lints and passes its + tests in the stage, its samples compiled in the snippet project; a planted compile error is + reported at the entity whose declaration holds it, through `Artifact.Spans`; a missing or old + toolchain skips that target's checks and says so, never reading as a pass; and a failing stage + leaves the destination untouched unless the caller lets the failure through + ([verification](../verification.md#after-the-output-builds)). +43. **T-43 · The manifest checked** — every operation and type the plan emits has exactly one + manifest entry, a dropped and a doubled entry each an error; every symbol the manifest names + exists in the built output, a planted rename that skips the manifest reported + ([verification](../verification.md#after-the-manifest-says-what-the-output-holds)). +44. **T-44 · Severity and versions** — each row of the seeded severity table, on every target and in + both channels; an enum gaining a member read in a response is behavioral breaking and, on Go, + structurally additive as apidiff agrees; Go's structural verdicts equal apidiff's over the + surface corpus; and a regeneration's bump follows the rule, a moved step version that changed an + artifact a patch and one that changed nothing none + ([surface verification](../surface-verification.md#the-version-a-regeneration-takes)). + diff --git a/docs/emitters/verification.md b/docs/emitters/verification.md new file mode 100644 index 000000000..f8f9e94e0 --- /dev/null +++ b/docs/emitters/verification.md @@ -0,0 +1,99 @@ +# Verifying a generation + +The mandatory tests pin bytes and behavior ([mandatory tests](testing/mandatory-tests.md)), and a +golden that does not compile is a golden nonetheless. So the engine checks a generation at three +points: before it, that each target honors what the document uses; while it runs, which versioned +steps shaped it; and after it, that the output builds and the manifest says what the output holds. +Each finding is a coded diagnostic whose severity the engine decides (INV5), so a check never fails +a run the caller did not ask it to fail. + +## Before: what a target honors + +A target that does not implement pagination, long-running operations, open enums, event streams, +file arrays in multipart or cookie parameters finds out construct by construct inside `Generate`, +and the user only after generating. So the engine compares, before generation, what the document +uses with what the target declares: + +- **What the document uses** is `irfacts.Rows`, derived on demand and never stored, as every + `irfacts` fact is ([derived facts](../ir/derived-facts.md)): each row of the [capability + matrix](../ir-spec-matrix.md) the document exercises, keyed by the row's stable key, with the + locations that exercise it. A test holds the detectors to the matrix's keys, as the conformance + corpus is held to them, so a row added to the matrix has a detector or a stated reason for none. +- **What the target honors** is `Capabilities.Rows` ([the registry](contract/registry.md)), the + keys it lowers. A row it degrades with a code (T-7) is not honored. +- **The difference** is reported per target and per row, a warning + `emitters//unhonored-row` naming the row and its first locations, before `Generate` + runs. The same two sets give a target's support matrix for its docs, generated rather than kept. + +## During: steps carry versions + +A generator changes between releases, and a user regenerating needs to know whether their SDK +changed, why, and what that does to their version number. A digest of the emitter says that +something changed and moves every SDK, whether or not the change touched anything it uses. So each +refine step and each boilerplate template carries a version, an integer bumped whenever what it +produces changes for some input, held by the step-trace golden as `plan.Revision` is held by the +plan's ([the plan's version](plan/overview.md#the-plans-version), T-10): + +- the step trace of a run says which steps fired, and the manifest records each one's version + ([what the manifest records](write-integrate.md#what-the-manifest-records)); +- a regeneration's release notes are the changelog entries of the steps it used whose versions + moved, and a moved step that changed an artifact sets at least a patch bump + ([surface verification](surface-verification.md#the-version-a-regeneration-takes)); +- a construct no step lowers is a T-7 diagnostic, as it always was. + +## After: the output builds + +`Generate` performs no I/O, so building what it returned is an engine stage of its own, run over +the generated files in a staging directory before the writer moves them into place. A generation +that fails it therefore never lands in the user's tree unless the caller lets that failure through. +Each target declares how its output is checked, in `Capabilities.Verify`: + +```go +type Verify struct { + Toolchains []Toolchain // what the checks need installed + Checks []Check // run in order, in the staged output root +} + +type Toolchain struct { + Name string // "go", "gradle" + Probe []string // the version probe, such as ["go", "version"] + Version string // a pattern capturing the version from the probe's output + Minimum string // the oldest version the generated code supports + Install string // guidance shown when it is missing or older +} + +type Check struct { + Name string // "build", "lint", "test" + Command []string // such as ["go", "vet", "./..."] + Needs []string // the toolchains it runs on + Reports string // how its messages give a position: "go", "kotlinc", "gradle" +} +``` + +- **A toolchain missing or too old** is reported with its install guidance, and that target's checks + are skipped and said to be: a check that could not run never reads as one that passed. +- **A failure maps to its entity.** The printer records which lines each top-level declaration + spans, `Artifact.Spans`, so a compiler message's file and line resolve to the entity whose + declaration holds it, and the diagnostic carries that entity's IR provenance, not just a path in + the output. +- **Samples are built too.** The usage samples each SDK target returns + ([docs and mocks](docs-and-mock.md#samples-come-from-the-sdk-emitters)) are compiled in a snippet + project the stage writes beside the SDK and never ships, so a sample the docs show is one that + builds against the SDK it documents. +- **CI runs the stage over every golden output**, so a golden that does not build fails the change + that produced it. + +## After: the manifest says what the output holds + +A manifest that is incomplete, or that names a symbol the built output lacks, is worse than none: +write/integrate, surface verification and every consumer mapping operations to SDK calls trust it. +It is built from `Origin` rather than inferred from source, so it is checked exactly: + +- **Coverage.** Every `OpID` and `TypeID` the plan says a target emits has exactly one manifest + entry, and every entry names an entity of the document. A miss or a duplicate is an error + diagnostic of the target's, since the manifest is the target's to build, with no tolerance. +- **Resolution.** After the build, every symbol the manifest names exists in the built output. The + check is the per-target extractor that [surface verification](surface-verification.md) reads an + SDK through, so it is that machinery's smallest use. A symbol missing, as a collision rename that + never reached the manifest or a keep region shadowing a generated symbol leaves one, is an error + diagnostic naming the entry. diff --git a/docs/emitters/write-integrate.md b/docs/emitters/write-integrate.md index e6e889ff7..6f6852d77 100644 --- a/docs/emitters/write-integrate.md +++ b/docs/emitters/write-integrate.md @@ -4,14 +4,16 @@ Post-milestone (architecture.md §2.3), but the contract is designed so it bolts change** (INV9). The `manifest.Manifest` and `Artifact` fields returned from `Generate` are the seams; the `Origin`-tagged AST makes the manifest a byproduct of the emit walk. -- **Generation manifest keyed by IR stable IDs.** Records spec/plan/emitter/config **hashes**, a +- **Generation manifest keyed by IR stable IDs.** Records what the generation was built from, a **sorted** file list (deterministic diffs), and an **entity → generated-symbol map keyed by `TypeID`/`OpID`/`PropID`** — assembled from the `Origin` on every emitted node. This is the - biggest win of stable-ID keying: a spec rename is "same ID, new symbol" — a lookup, not oagen's - structural rename-inference machinery. A declaration derived from an entity keys under it, as - `User.ID#create` does ([`User`](targets/go/models.md#user-in-go)), a union's wrappers and fallback - among them, so no two symbols share a key. Write/integrate and surface verification - (architecture.md §2.3) both rest on it. + biggest win of stable-ID keying: a rename of anything an ID is not built from is "same ID, new + symbol" — a lookup, not oagen's structural rename-inference machinery — and a rename of what it is + built from, a path or a component name, is carried by the correlation key each entry records. A + declaration derived from an entity keys under it, as `User.ID#create` does + ([`User`](targets/go/models.md#user-in-go)), a union's wrappers and fallback among them, so no two + symbols share a key. Write/integrate and surface verification (architecture.md §2.3) both rest on + it. - **File-header provenance gates pruning.** `Artifact.Header` writes a generated-by header; the writer **never deletes a file lacking it** (hand-edited). First adoption (no prior manifest) skips pruning, writes a baseline, emits a notice. @@ -24,3 +26,130 @@ seams; the `Origin`-tagged AST makes the manifest a byproduct of the emit walk. - **Staleness check.** `(prior-manifest entities − current entities) ∩ files-on-disk` surfaces dead code additive writers cannot prune — cheap because both manifests are ID-keyed and the prior IR/plan snapshot is serializable (INV7). + +## What the manifest records + +A manifest holds the choices a later generation must honor and cannot recompute from the spec: the +names it shipped, the behavior it pinned, the files a user now owns. So it records more than hashes: + +```go +package manifest + +type Manifest struct { + Schema int // this format's version; an older one is migrated forward + Target string // the emitter's TargetKey + Options int // OptionsVersion(), the behavior pinned for this SDK + // (contract/options.md) + Inputs Inputs // what the generation was built from + Files []File // every file written, sorted by path + Entities map[EntityID]Entry // every emitted entity and each declaration derived from one + Steps map[string]int // each refine step and template that ran → its version + Regions []Region // the hand-written regions found, each by its entity (below) +} + +type Inputs struct { + Doc string // the document's digest, which Plan.Version takes + // (plan/overview.md) + Plan string // Plan.Version + Emitter string // the emitter's own version + Config string // SHA-256 of the decoded options, resolved policy and resolved + // hints, each as its canonical JSON +} + +type File struct { + Path string + Digest string // SHA-256 over the bytes written, normalized (below) + Origins []EntityID // as the Artifact lists them + Once bool // written when absent, the user's thereafter +} + +type Entry struct { + Key string // the entity's correlation key, from the pass #679 plans + Unit string // the unit the symbol was placed in + Base string // its rendered name before a collision was resolved + Symbol string // the name the generation gave it (refine/naming.md) + Kind string // the target-AST node kind it was declared as + File string // the path it was written to +} +``` + +- **The document, not the spec text.** `Inputs.Doc` is the IR's digest, canonical JSON with + `sources` left out. Reformatting a spec, or moving it to another path, changes the spec's bytes + and leaves the IR whole, so it is no new generation. +- **A file as written.** `Digest` covers the bytes after formatting, with a byte-order mark + stripped, line endings made `\n` and trailing newlines dropped, so a formatter run or a checkout + that converts line endings is not mistaken for an edit. A file carrying the generated header + whose bytes on disk no longer match was edited outside its regions. The writer leaves it as it is + and reports it as a coded diagnostic, whose severity the engine decides, rather than overwrite + the edit. +- **Nothing to do.** When every `Inputs` digest and every file's digest match the prior manifest, + the engine writes nothing. +- **Behavior and steps.** `Options` is the version whose defaults this SDK keeps until its owner + moves it ([target options](contract/options.md)), and `Steps` names the version of each step + that shaped it, which [surface verification](surface-verification.md) reads to say what a + regeneration changed and why. +- **Written once.** A file a target generates for the user to own, such as a test to fill in, is + `Once`: written while absent, never rewritten or pruned. It is matched to a later generation by + its origins' correlation keys, so a renamed entity's file is not written a second time under the + new name. +- **Its own version.** Unlike the IR, a manifest holds choices that cannot be recomputed, so an + older `Schema` is never refused: each version step has a migration, applied in order, and a + manifest from a newer Morphic is refused, since its fields cannot be read. + +## Regions follow their entity + +A `@morphic:keep ` region belongs to an entity: the one whose declaration encloses it, or, at +a file's top level, the one whose declaration it follows, else the file's first origin. The +manifest records each as a `Region`, keyed by that entity's ID, its correlation key and the region's +name, with the region's text. So a region follows its entity: when a rename moves the entity to +another file, the next generation writes the region after the entity's declaration there, and the +old file, emptied of generated symbols, is pruned as any other. + +```go +type Region struct { + Entity EntityID // the entity it belongs to + Key string // that entity's correlation key + Name string // the name its marker gives it, unique within the entity + File string // where it was found + Text string // what it held, markers excluded +} +``` + +A region whose entity no longer exists is reported and kept: its file is not pruned while it holds +one, and the staleness check lists it. Merging stays additive. A mode that rewrote generated code a +user edited would take ordinary three-way merge semantics, the prior generation as the base and +conflicts written with the markers merge tools read, never a merge of its own invention. + +## A key that matches nothing + +Every input keyed by an IR ID can outlive its entity: a hint ([shaping +hints](contract/shaping-hints.md)), a runtime-policy override +([layers](contract/runtime-policy.md#layers-and-operations)), a per-target naming overlay, an +overlay, the prior manifest. A spec change that removes or renames the entity leaves the key +matching nothing, and ignoring it would drop the user's customization in silence. So every such key +is reported, as a warning `emitters/unmatched-key` whose severity the caller controls and which +never aborts the run. It names the key and, where one exists, the entity whose correlation key +equals the one recorded for it, as the likely new home. A prior manifest's entry for a removed +entity is not such a key: the staleness check reports it. + +## A document the user writes around + +A README is mostly hand-written, with generated sections: installation, authentication, an +example per feature, a table of contents. A keep region protects hand-written islands in generated +code; a README is the reverse, so it is integrated as blocks. The docs emitter +([docs and mocks](docs-and-mock.md)) returns each generated section as a block with a stable ID, +between `` and ``, in an artifact whose +`Integrate.Blocks` is set. The engine's integrate step then rewrites only those blocks in the file +on disk: + +- a block found there is replaced by the new one of its ID, and everything outside the markers is + kept byte for byte; +- a block marked `` is the user's, and kept as it is; +- a block new to this generation goes after the last block it follows in the generated order, or + at the end; +- a block the generation no longer has is removed, unless frozen, and reported; +- the table of contents is a block of its own, rebuilt last from the headings of the merged file, + so it lists the user's sections too. + +A file with no markers yet takes the generated blocks after its own text, once, and a first +generation writes the whole artifact. diff --git a/docs/ir/derived-facts.md b/docs/ir/derived-facts.md index 94ed246b7..f3aa1fcda 100644 --- a/docs/ir/derived-facts.md +++ b/docs/ir/derived-facts.md @@ -90,3 +90,14 @@ value of exactly that type can hold, at any depth, one of the same shape — and the types whose shapes share its strongly connected component. A reference to a polymorphic base counts every subtype it may hold, so a subtype holding its own discriminated base is recursive. A cycle through composition alone, which the IR permits, is reported too. + +## Capability rows + +`Rows()` answers which rows of the [capability matrix](../ir-spec-matrix.md) a document exercises: +for each row's stable key, the locations of the nodes that exercise it, sorted, such as every +operation whose `Pagination` is set for `pagination`. One detector per row reads the facts the row +names, and a test holds the detectors to the matrix's keys, as the conformance corpus is held to +them: a row added to the matrix has a detector, or a reason no IR fact shows it. The engine reads +`Rows` to tell a user, before generation, which rows a target does not honor +([verification](../emitters/verification.md#before-what-a-target-honors)). + diff --git a/docs/ir/identity.md b/docs/ir/identity.md index b03f8a463..a824002d7 100644 --- a/docs/ir/identity.md +++ b/docs/ir/identity.md @@ -10,11 +10,12 @@ type ServiceID, GroupID, ChannelID, MessageID, AuthID, PropID, ParamID, EnumMemb type ServerID string ``` -IDs are opaque to consumers but constructed deterministically by compilers from the source -pointer of the defining occurrence, so the same input always yields the same IDs (stable -snapshots, cachable, diffable across spec revisions). IDs are never derived from display names -and never rewritten by renames. The `dedup` pass may alias two structurally identical anonymous -types; aliases are recorded so both IDs stay resolvable. +IDs are opaque to consumers but constructed deterministically by compilers from the source pointer +of the defining occurrence, so the same input always yields the same IDs (stable snapshots, +cachable, diffable across revisions that keep each entity's pointer; one that renames a path or a +component gives what is beneath it new IDs, #419). IDs are never derived from display names and +never rewritten by renames. The `dedup` pass may alias two structurally identical anonymous types; +aliases are recorded so both IDs stay resolvable. ## The ID grammar diff --git a/docs/prior-art.md b/docs/prior-art.md index 85681386d..8d6cf796d 100644 --- a/docs/prior-art.md +++ b/docs/prior-art.md @@ -44,7 +44,8 @@ and the differ/compat/verify subsystems added since the original survey). symbol map), file-header provenance that gates pruning (never delete a file lacking the generated-by header), ignore-region markers for hand-written islands, and additive-only AST merges. Manifests should key entities by stable ID — oagen's `"METHOD /path"` keys break - on path renames. + on path renames. So do pointer-derived IDs, which is why a correlation key is planned beside + them (#679). - **Wire-conformance smoke testing with a spec-only offline baseline** — the expected request shape (method, path, query, body keys) is derived from the spec alone and diffed against the generated SDK running under HTTP interception; request-side mismatches block, response-side @@ -89,7 +90,8 @@ and a self-correcting overlay loop. Three structural lessons: model even has a dual-key design (`id` + name) intended for rename detection — but `id` is derived from the name, so the rename branch is dead. Cross-language change rollup finally reinvents a spec-level identity (`conceptualChangeId`). Morphic's pointer-derived IDs make - rename detection a lookup; the compat experience adds one refinement — when both pointer + detecting a rename that keeps the pointer a lookup; the compat experience adds one + refinement — when both pointer *and* name churn, a third *structural* correlation tier (content-hash / field-set similarity, shared with the `dedup` pass) is the fallback (see [open question 5](./ir/open-questions.md)). diff --git a/docs/reference-learnings.md b/docs/reference-learnings.md index 6f3e4da5c..157f08e85 100644 --- a/docs/reference-learnings.md +++ b/docs/reference-learnings.md @@ -178,7 +178,9 @@ touches some slice of it: if its content starts with the auto-generated header*, else preserve + report as hand-edited; **skip pruning entirely on first adoption**. Adopt this schema directly. Morphic's one improvement (already specced): key the entity→symbol map by **IR stable ID**, not oagen's `"METHOD /path"` - string, so it survives path/name churn. Record `service` per op (oagen needs it for scoped runs). + string, so it survives a rename of anything the ID is not built from, and record a correlation + key (#679) beside it for a path or name rename. Record `service` per op (oagen needs it for scoped + runs). - **Scoped regeneration NEVER filters the spec** (oagen `orchestrator.ts:37`, `generate-files.ts:44`; fastapi `--specify-tags`; openapi-python full-rmtree). Placement/dedup/shared-files/barrels are computed over the *full* (or `selected ∪ already-on-disk`) surface so shared files stay byte-