Repository navigation
feat(irfacts): derive a correlation key for every entity - #1354
Open
OmarAlJarrah wants to merge 64 commits into
Open
OmarAlJarrah wants to merge 64 commits into
OmarAlJarrah wants to merge 64 commits into
Conversation
An anyOf lets one value satisfy several variants, and the emitter design lowered every untagged union to a closed sum that holds exactly one. Where two variants admit the same value, holding the first misnames it and drops the members only the others declare. The plan now proves, once per union, whether two variants of an anyOf can admit one value (TypePlan.Inclusive), by JSON kind, by finite value sets and by a member one model requires that the other cannot hold. A union it proves disjoint lowers as a closed sum, as a oneOf does. One it cannot is inclusive: it holds an optional member per variant, set for each variant the value admits, decodes into every candidate variant, and encodes the merge, a disagreement being an encode error. A null a nullable variant admits is held by the reference, never by a variant. Go lowers an inclusive union to a struct with a pointer per variant and Kotlin to a class with a nullable property per variant. T-29 pins the round trip of a value two open models admit.
A positional model is an array on the wire, so the kinds rule now says so and the member rule reads only models laid out by name. A stream that ends on a terminal variant now ends on an element holding one, among the several an inclusive union holds. A null literal variant now lowers as a nullable variant does, so either spelling of a null member reaches the reference. Inclusive joins the plan fields the plan-or-refine test keeps, and the docs emitter says when a value may be several variants at once.
plan.Policy held a function, PrimaryResponse, and Plan.Version hashed the policy. A function has no stable bytes: json/v2 refuses it, its printed form is an address that changes every run, and skipping it gives two policies that pick different responses one version. The policy is now data from a closed vocabulary: PrimaryResponse is an ordered list of rules and DefaultFails says whether an unpicked Default is the catch-all failure. Version hashes the document's digest without sources, a new plan.Revision bumped with every change to Build's output, and the policy's deterministic JSON. EnvelopeDetect named a type nothing defined. plan.Policy.Envelopes now lists envelopes a configuration names, none by default. Each matches a body by the bases it reaches or by the members it declares, and says where the value sits and which flag value means the call failed. A matched success returns that member, keeps the envelope reachable through WithResponse, and turns a 2xx whose flag says it failed into a failure with no fault or kind, decoded as the operation's matching error body. Failures and pages are read whole. T-30 pins the plan's identity and T-31 the envelope reads.
…into docs/emitters-envelopes-policy-data # Conflicts: # docs/emitters/docs-and-mock.md # docs/emitters/testing/mandatory-tests.md
Runtime policy was one tree per project, so an operation needing its own timeout or retry had nowhere to say so, and a document's own SDK configuration, kept verbatim as vendor extensions, would have been parsed by each emitter its own way. Policy and hints are now layers, each holding a document-wide override and overrides by operation: what a document embeds, the project's, the emitter's. A setting takes the highest layer that sets it, an operation's value over its layer's document-wide one, and a declared IR fact beats every layer. A new emitters/embed reader turns vendor extensions a configuration maps into the lowest layer of each, decoding each value strictly and reporting what it cannot read. The vocabulary gains connection-error retries, Retry-After, a throttling backoff and redirects. Whether an operation is safe or idempotent is a plan fact, OpPlan.Effects: declared, else from its HTTP method. An enum a client reads keeps a value it does not list, as a union does, and a oneOf value two variants admit decodes as plan.Policy.ExclusiveMatch says. A service hint makes a parameter a client setting; nothing is hoisted by inference, since a shared parameter stops being shared as an API grows, and the plan only reports candidates.
An enum whose value type has no Go constant form, a date, a uuid or a decimal, had no Go lowering, and the type table built an enum value by decoding its wire form, which no constant can be. Kotlin's enum class writes every member as its @SerialName string, so a boolean or numeric enum went out as a string its schema does not admit. A member is now written as the JSON kind its value type has, and decoded by comparing as JSON compares values, numbers by value. Go declares an enum over its value type's row, and over the wire text where the row has no constant, the codec writing the value type's JSON kind and a Value method giving the row's type. Kotlin generates a KSerializer for every enum that is not a string enum, and an entry of a mapped type holds its value as that type. T-36 pins every value type on every target.
Kotlin's naming document stated only the members it gives every type, so a Kotlin emitter had no names to reserve for a group's module or for the declarations a unit holds beside the spec's, and T-19 no fixture. Measured with Kotlin 2.4.0 and Gradle 9.8.2: a module named build sits in the root project's output, which clean deletes with the module's sources; one named buildsrc is taken for Gradle's buildSrc on a case-insensitive file system and fails the build; gradle and src share the wrapper's and the root sources' directories. Those four are the units set, and a group's module is placed at the build root so the set applies. Hard keywords need no reservation: backticked as package segments they compile on the JVM, JS and Native and link into a framework, and kotlin is refused only as the first segment, which the options name. The unit set is Client, a group's sub-client and the runtime names the root unit declares.
The manifest recorded hashes, a file list and the entity-to-symbol map, too little for a regeneration to honor what the last one did: an edited generated file was overwritten, a no-op rewrote every file, a reformatted spec looked new, and keep regions were found by path. The manifest now records the document's digest instead of the spec's, a normalized digest per written file, the options version, the steps that ran, files written once, each entity's correlation key, base name and symbol, and each keep region by its entity, under a schema version migrated forward. Regions follow their entity to its new file. Every ID-keyed input that matches nothing is reported with its likely new home. A README is merged as blocks the user writes around. Collisions now read the prior manifest: a shipped symbol keeps its name while its ID and correlation key agree, and only newcomers take a suffix, so adding an endpoint renames nothing. The prior manifest joins the inputs the purity rule names. A spike at Go 1.27.3 chose dst for the Go merge adapter: go/ast nodes moved between files lost their doc comments, one into a function body, while dst and a text splice were both correct, and only dst reaches inside a declaration. Claims that IDs survive any rename are narrowed to renames that keep the pointer.
The docs emitter reused the SDK's primary response and content, had no design for URLs, links, collisions or versions, spelled usage examples it could only re-derive, and left documentation extensions and agent documents unaddressed. Every artifact would also synthesize its own example values. The docs now render every response, error case and content, the primary only ordering them, and each type's request and response shapes apart. A route plan derives every URL from an ID-keyed catalog through templates, refuses any two entities on one URL after the templates apply, resolves links as ID requests, and keeps one entity's identity across version snapshots through route tables the engine passes in. Agent Markdown and llms.txt share those routes and escape spec text. SDK emitters return a usage sample per operation for the docs, and the plan holds one set of examples, declared first and otherwise seeded from each entity's correlation key, for docs, samples, tests and mocks. Documentation extensions are read by mapping, internal markers left to the filter pass, and a CLI is planned as an extension point built from IR facts.
EmitterInput.Policy became policy.Resolved when policy gained layers and per-operation overrides, but the refiner context still held a single policy.Policy, so a step had no way to read an operation's own settings. Ctx.Policy now holds the resolved policy the emitter is given.
…o docs/emitters-enum-value-types
…into docs/emitters-kotlin-reserved-sets
…ts' into docs/emitters-generations
Collision resolution now keeps the names the prior generation shipped, read from its manifest, but the refiner context carried no prior state, so the naming step had nothing to read it from. Ctx.Prior carries the PriorState the emitter is given.
…docs/emitters-docs-artifacts
The docs emitter now receives every SDK target's usage samples and every version snapshot's route table, but the shared refiner context carried neither, so a docs refine step could not read them. Ctx holds both.
Nothing required generated output to build, a target found the constructs it could not lower one at a time inside Generate, a regeneration could not say what changed or why, nothing checked the manifest, and surface verification's records lacked the facts its verdicts turn on. The engine now checks a generation at three points. Before it, the capability-matrix rows a document exercises, a new derived fact, are compared with the rows a target declares, and each one unhonored is reported. During it, every refine step and template carries a version the manifest records. After it, a verify stage builds, lints and tests the staged output and compiles the usage samples, mapping each toolchain message to the entity whose declaration holds it, and the manifest is checked for exact coverage and for symbols that resolve in the build. Change records now carry direction, response optionality and bindings, the seeded severity table gives one verdict per channel, structural for callers' code and behavioral for deployed SDKs, and the version bump follows from them. apidiff is the test oracle for Go's structural verdicts, neither the differ nor the severity.
The rule saying what a oneOf decodes a value two variants admit to landed at the end of the section on a variant's null, which it has nothing to do with. It belongs with how a variant is recognized, which is also where T-35 links it.
…o docs/emitters-enum-value-types
…into docs/emitters-kotlin-reserved-sets
…ts' into docs/emitters-generations
…docs/emitters-docs-artifacts
…to docs/emitters-verification
A catalog, a linter or a diff reads the IR as an input format, and the design documents answered its questions only for emitters. This states the answers where such a consumer will look: - identity across revisions: an ID holds only while its pointer does, and an array index can hand an old ID to another entity, so matching across revisions rests on a correlation key derived from what the document names an entity by (designed in derived-facts.md); - what a key in Unmodeled promises: a vendor extension's name is the source's, every other part of a key is the compiler's spelling, and a change to it moves IRVersion as a renamed symbol does; - what to pin, and when a stored document must be recompiled; - why the compiler has no provenance-trimmed mode, and that a slice of a document is the planned filter pass. Open question 5 now records the answer and leaves only the structural tier open. The emitter documents cite the new sections in place of the issues they close.
Building the correlation keys against the golden spec showed three rows of the design wrong. Servers there share URL templates, so a server is keyed by the scope naming it as well. A header carries a property ID of its own, so it is an entity to key, not only a type. And two declarations of one wire name in one composed model would leave both unkeyed, though the flattening takes one, so the key follows the flattening. A variant has no ID, so the table now keys its type rather than the variant.
An ID is a function of where an entity sits in the source, so it holds across revisions only while that position does, and an array index can hand an old ID to a different entity. The golden spec's own history has one: the organization accounts listing's tenth parameter schema is the direction parameter's at one revision and the status parameter's at the next. irfacts.Index.Correlation now answers what an entity of any class is called across revisions, and Correlated the reverse, as docs/ir/derived-facts.md designs it. A named entity is keyed by its name. Everything else is keyed by the least key among the positions naming it, settled in key order as a shortest-path search settles the nearest vertex. A key two entities share keys neither and nothing through them, since a wrong match is what the key exists to prevent. On the golden spec every operation, parameter, group, server and auth scheme gets a key, as do 91% of types, 96% of properties and 92% of enum members. All but two of the rest sit beneath a union variant nothing names. Deriving every key takes about 200 ms.
5 of 6 tasks
4 of 5 tasks
…o docs/emitters-enum-value-types
…into docs/emitters-kotlin-reserved-sets
…ts' into docs/emitters-generations
…docs/emitters-docs-artifacts
…to docs/emitters-verification
… docs/ir-consumer-contract
…feat/irfacts-correlation-keys
5 of 6 tasks
Promoted fields in the test literals, preallocated slices and a slices.Backward loop, as modernize and prealloc ask.
The modernize linter's embedlit check flags each type literal that names its embedded TypeCommon; the fixtures now set ID, Name and Anonymous directly, as the rest of the module does.
17 tasks done
…into docs/emitters-envelopes-policy-data
…ata' into docs/emitters-policy-layers
…o docs/emitters-enum-value-types
…into docs/emitters-kotlin-reserved-sets
…ts' into docs/emitters-generations
…docs/emitters-docs-artifacts
…to docs/emitters-verification
… docs/ir-consumer-contract
…feat/irfacts-correlation-keys
This was referenced Nov 2, 2026
This branch stood on docs/emitters-verification (#1339), the top of the emitter design stack, because five of its edits rewrote lines #1336 wrote. Its own changes need nothing from that stack, so this merge takes main's side of every file the stack changed and keeps only this branch's changes, applied to main: - The citations of #679, #678 and #419 it replaced in plan/overview.md, refine/naming.md, surface-verification.md and write-integrate.md exist only on #1336's branch, so those edits drop out. #1336 can link the new sections once this lands. - identity.md's opening paragraph is rewritten from main's wording, not #1336's, and "Across revisions" no longer cites the prior manifest's section, which only #1336 adds. - The correlation keys section ends derived-facts.md, which on main has no capability rows section before it, and the README row no longer names one. - The #429 citation in embedded-policy.md is replaced in main's wording of the paragraph.
#1327 moved a path item's own summary and description to openapi:pathItem/summary and openapi:pathItem/description, at IRVersion 0.30.0. Compiling a path item that declares both, with two operations, puts both keys on each of them, which is where the table now names them.
Its base left the emitter design stack and stands on main, so this branch now builds on main too. The citations of #679 this branch linked in architecture.md, prior-art.md and reference-learnings.md exist only on #1336's branch, so those edits drop out, and the two correlation entries join main's list of reachability roots.
This was referenced Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An ID is a function of where its entity sits in the source. Across two revisions it therefore holds only while that position holds, and an array index can hand an old ID to a different entity. The golden spec's own history has a case. At revision b7e65ddd,
t/anon/paths/~1organizations~1{organization_id}~1accounts/get/parameters/10/schemais thedirectionparameter's schema; at 319dc756 it is thestatusparameter's. A consumer keying state by ID, such as an emitter's prior manifest, would give one the other's name.This builds the correlation key that
docs/ir/derived-facts.md#correlation-keysdesigns (the design lands in #1351, this PR's base):irfacts.Index.Correlation(id)returns what an entity of any ID class is called across revisions.Correlated(key)returns the reverse, so matching two revisions takes one call on each side.operationId, else its method and route with variables erased (a webhook by its event name). A type by its namespace and name, a primitive by its kind, and a service or auth scheme by its name. A group is keyed by its ID, which is never built from a position.irfactsfact is.Measured on the golden spec (spec @319dc756, IR from morphic @b206dc99):
Building against the golden spec corrected three rows of the design, which went back onto #1351. Its servers share URL templates, so the scope keys a server too. A header carries a property ID of its own. And a wire name declared twice in one composed model keys the declaration the flattening takes, rather than neither.
The consumers page (now "carried"), the package lists in
CLAUDE.mdandREADME.md, and the emitter contract'sFactscomment now link the section.architecture.md,prior-art.mdandreference-learnings.mdcite #679 only on #1336's branch, so #1336 can link them once this lands.Closes #679
The second commit replaces a local wire-name helper with
ir.Property.EffectiveWireName; its subject is 78 characters with this PR's number appended. A later commit spelling the test fixtures with promoted fields is 74 with it. The branch has several commits, so the PR lands under its title, which fits.Stack
Base: #1351, which targets
main. Merge #1351, then this PR; GitHub retargets this PR tomainonce #1351 lands.Test plan
go test ./ir/irfacts -run TestCorrelation: each named class, every position class, the depth bound on groups, callbacks under named and unnamed parents, in-place conjuncts, all three variant naming rules, a shared key, and the least-position rule in both declaration orders.morphic compileoutput. The probe also caught headers being keyed only through their types.main(e9f1707 merges it):go build ./...,go test ./ir/irfacts, and archtest's reachability, layering, method-cap and document tests pass. The two IR fieldsmainadded since,FileInfo.InferredandPartEncoding.Inferred, hold no type, so no position goes unkeyed.mainor the base,skills_lintclean; the two subjects past 72 characters with the number are named above.apiRootsnames the two methods), and coverage.