Repository navigation
core: the four core seams have no contract suite a third-party provider can run #342
Description
Context
Part of #56. Found by the 1.0 surface audit (#319), against main @ 550a41ef.
#56's definition of done is a third-party provider, written against the
previous minor, that "still compiles and still passes the interface's contract
test". Of the nine documented seams, only AgentBackend has a contract test a
third party can run (runBackendContract, @schlessera/brain-ui-sdk/testing).
The four core seams have none:
| Seam | Declared | What the seam promises beyond its type |
|---|---|---|
EmbeddingProvider |
packages/core/src/lib/seams.ts:54 |
dimensions matches every vector returned; embedQuery honours or tolerates opts.signal (added in cc48069); an absent embedImage / embedPdf degrades to the text description (docs/extending/README.md:50-58) |
CompletionProvider |
seams.ts:22 |
capabilities.vision: false means image parts are not sent, and core falls back to title-only descriptions |
AgentRunner |
seams.ts:37 |
capabilities.streaming agrees with whether runStreaming exists; timeoutMs is honoured |
SkillEmitter |
seams.ts:97 |
the emitted layout for a SkillManifest (seams.ts:79) |
The promises are written in docs/extending/embeddings.md, completions.md,
agent-runners.md and skill-emitters.md. Only the built-ins are tested
against them, in files a third party cannot import:
packages/core/tests/gemini-query.test.ts, claude-runner-*.test.ts,
skills-*-emitter.test.ts. The promotion bar (docs/extending/README.md:191-198)
asks a community provider for "keyless, deterministic tests that exercise the
runtime" and points to gemini-query.test.ts as the model. It cannot point to
a suite, because there is none.
Scope
- Add a
./testingsubpath export to@schlessera/brain, shaped like
@schlessera/brain-ui-sdk/testing: test-runner primitives are injected,
there is no runtime dependency on a test runner, and nothing needs a key or
the network. - One suite per core seam, asserting the promises in the table above and in
each seam's page underdocs/extending/. Where a page's promise is too vague
to assert, record that on this issue rather than inventing a stricter one. - Run every built-in through its suite in this repository's tests:
gemini
embeddings,gemini-flashandanthropic-haikucompletions, the built-in
agent runners, andBUILTIN_EMITTERS. Use isolatedfetchor fake
binaries, as the existing tests do. - Point the promotion bar and each seam page at its suite.
Out of scope
- Changing any seam's interface or a built-in's behaviour. A built-in that
fails its suite is a bug, filed on its own. AgentBackend(it has a suite, extended by ui-sdk: the AgentBackend contract suite never exercises a deny or the turn posture #341) andSiteAdapter(scrape: SiteAdapter has no implementation in the tree; the ten job boards use module-jobs interface #344).DocumentRenderer. Whether it is a seam is docs: decide what 1.0 freezes where the contract document does not say #343, question 3.
Acceptance criteria
-
@schlessera/brain/testingexports one suite per core seam, and the
packaging smoke test imports it from the built package. - Every built-in listed under Scope runs through its suite in CI, with no
key and no network. - Each suite has at least one case shown failing against a deliberately
non-conforming fake, for example an embedder whose vectors are not
dimensionswide. The mutation and the failing assertion are recorded
in the PR. -
docs/extending/README.md(promotion bar, item 3) and the four seam pages
name the suite.
Verification
bun run test; bun run build and the packaging smoke test; the mutation
receipts in the PR.
Contract impact
- This touches the integration contract, so it needs the
contractlabel, aCONTRACT:commit prefix, and the doc updated in the same commit. - This changes a published package's behaviour, so it needs a changeset.
A new subpath export is additive. A new export must also be added to
api-report/core.txt (bun run api-report).
Activity
schlessera commentedon Sep 25, 2026
Promises on the four seam pages that were too vague to assert, recorded here as the Scope asks, rather than turned into stricter checks in
@schlessera/brain/testing:All four seams
id(andSkillEmitter.agent) is only typedstring. No page says it must be non-empty, although core puts it in composite ids and in the vector-table metadata. The suites check the type only.EmbeddingProvider.dimensionsis only typednumber. No page says it is a positive integer; the suites check that every vector matches it, not what it is.EmbeddingProvider (
embeddings.md)embedreturns vectors in the order of its texts. The indexer relies on it (it zips vectors back onto chunks), but the suite checks count and width only.embed([])returns.opts.signal" is two promises. The suite checks the tolerant half for every provider. It checks the honouring half only when the harness supplieshanging(), which declares that the provider forwards the signal.CompletionProvider (
completions.md)systemandmaxTokensmust be honoured, or only accepted, is unstated. The suite checks that they are accepted.vision: truesays a provider "can look at image/PDF parts". That cannot be observed through a fake vendor. The suite checks that core then hands it the part, and that its answer becomes the description.AgentRunner (
agent-runners.md)rundoes when the agent fails (exit non-zero): reject, or resolve to the text it has. The built-ins reject. The suite does not check it.capabilities.skills("the agent discovers the brain's skills") cannot be observed without a real agent. The suite checks the type only.runStreamingtakes notimeoutMs, so it makes no timeout promise, and the suite checks none.SkillEmitter (
skill-emitters.md)