Repository navigation
Description
Context
Part of #56. Found by the 1.0 surface audit (#319), originally against main @ 550a41ef. The defect description and source ranges below are verified against the pre-fix baseline dee9b10e; they describe the behavior before PR #661.
#56's definition of done is a third-party provider that "still passes the
interface's contract test". For AgentBackend that test exists and is
published: runBackendContract in @schlessera/brain-ui-sdk/testing
(packages/ui-sdk/src/testing/index.ts:93). It covers capabilities shape,
profile listing, the turn lifecycle, abort, failure, per-session busy-ness and
unknown profiles (:100-397). It does not cover the permission surface, and
the permission surface is where the seam has moved since 0.33.0:
- A deny is never exercised.
packages/ui-sdk/README.md:48-50says
"permissions: truemeans a deny actually blocks a mutation, and the shared
contract test suite enforces it". The suite's only permission code is a fake
bridge that always allows (testing/index.ts:72). A backend that declares
permissions: trueand runs the tool anyway passes. - The turn posture has no conformance test and no declaration.
StartTurnRequest.enforceAllowedTools(packages/ui-sdk/src/server/backend.ts:318,
added in ecc93b9) andnoGrantSurface(:342, added in 1de4d6c) each say
"A backend that does not understand the field ignores it, which is why a
[narrower] posture must also verify its backend honours it". No capability
says a backend honours either, so a host cannot check. Both first-party
backends are tested for it in their own suites
(packages/ui-backend-claude/tests/no-grant-surface.test.ts,
packages/ui-backend-pi/tests/grant-surface-posture.test.ts), which a
third-party backend cannot run. BackendModulehas no conformance test. The descriptor is at
packages/ui-sdk/src/server/backend-module.ts:153. Its context now accepts
confirmBashPatterns: readonly ConfirmPatternSource[] | null(:100). The
published suite has no descriptor profile parsing/resolution case a third-party
author can run to detect incompatible assumptions.costReportingcontradicts the cost rule.BackendCapabilities.costReporting
(backend.ts:64-65) says "result.costUsd is meaningful (0 otherwise)". The
contract (docs/integration-contract.md:911-915,:1150-1151) says a cost isnullor
absent when unknown, never zero.
Scope
- Add suite cases to
runBackendContract: a backend with
capabilities.permissions: truewhose bridge denies a mutating tool does not
run it, and emits the denial. - Maintainer ruling (2026-09-28, A): mandatory restriction handling by 1.0.
Every conforming backend must honor enforceAllowedTools and noGrantSurface
when requested, or reject an unsupported restricted turn before execution.
Silent ignoring is forbidden. No optional capability flags or harness-only
declaration. Optional request fields remain optional; ordinary turns retain
their existing defaults. Update the misleading ignore-unknown-fields comments. - Extend the shared suite to exercise both restrictions and safe rejection.
Enforcement means off-allowlist tools cannot bypass a permission decision;
noGrantSurface denies promptly without an unanswered approval request, and
remains invalid without enforcement. The two shipped backends must demonstrate
actual enforcement, not pass solely by rejecting every restricted turn. - Record this baseline and rejected alternatives in docs/decisions, align the
integration contract and authoring documentation, and explain that this is a
pre-1.0 tightening of conformance, not an immediate general seam freeze. - Add a
BackendModuledescriptor case: profile parsing and resolution through
defineBackendModule, run against both first-party descriptors. - Correct the
costReportingdoc comment to match the contract's cost rule.
Out of scope
- Changing what either first-party backend does. They should pass the new cases
as they are. A case they fail is a bug, and it gets its own issue. - Suites for the other seams. The core seams (core: the four core seams have no contract suite a third-party provider can run #342) and
SiteAdapter(scrape: SiteAdapter has no implementation in the tree; the ten job boards use module-jobs interface #344) have their own
issues.
Acceptance criteria
- The maintainer selected mandatory handling (A), recorded 2026-09-28.
- Source comments, integration/authoring docs and a durable decision record agree on mandatory handling, safe rejection and unchanged ordinary-turn defaults.
- Both first-party backends pass the extended suite through their actual adapter paths with keyless scripted runtimes; denied tool bodies provably do not execute.
- Off-allowlist operations require a permission decision; denied operations do not run. No-grant turns do not request an unanswered card; invalid posture combinations and unsupported restricted turns fail before execution. These cases are mandatory conformance coverage, not optional capability-gated tests.
- Each new case fails when its guard is removed: a fake backend that ignores
the deny, one that admits an off-allowlist tool under
enforceAllowedTools, one that parks undernoGrantSurface, and a
descriptor that rejects a valid profile. The mutation and its failing
assertion for each are recorded in the PR. -
packages/ui-sdk/README.md's claim about the suite is true as written, or
rewritten to say what the suite does.
Verification
bun run test; the mutation receipts in the PR.
Contract impact
- This touches the integration contract, so it needs the
contractlabel, aCONTRACT:commit prefix, and the doc updated in the same commit. - This changes a published package's behaviour, so it needs a changeset.
@schlessera/brain-ui-sdk/testing gains cases, so a third-party backend that
passed before may fail. That is the point of the issue, and the changeset says so.
Updated 2026-09-28: selected mandatory restricted-turn handling; refreshed source references, added durable record/documentation scope and concrete behavioral checks. Requires contract/breaking labels and a changeset naming stricter conformance requirements. Ready for implementation.
Updated 2026-09-30: verified and corrected source ranges against the explicit pre-fix baseline; PR #661 supplies the implementation and behavioral evidence. The original audit commit remains historical context.
Completed 2026-09-30 in #661, squash commit c849bf27433d2af82ba4fb5e2ec20f3cb7b71aab, verified on origin/main. The extended suite, descriptor cases, mandatory-handling record, aligned documentation and cost comment are delivered. Six restored behavioral mutations are recorded in the PR. Full isolated local suite: 7,566 pass, 45 skip, 0 fail; all 12 current-head PR checks pass. Separate findings #659 (unknown-cost lifecycle assertion) and #660 (doctor test isolation) remain open; neither is required unfinished work from this issue.
Activity
schlessera commentedon Sep 28, 2026
Decision (maintainer, 2026-09-28): Option A. Make handling of enforceAllowedTools and noGrantSurface mandatory for every conforming AgentBackend by 1.0, enforced by the published shared contract suite. Do not introduce optional posture capability flags or a harness-only opt-in. When a backend cannot uphold requested restrictions it must reject the turn safely before execution; it must never silently ignore them. These fields remain optional per-turn inputs; mandatory support does not turn the restrictions on for every ordinary chat turn.
Preserve existing semantics: enforceAllowedTools requires permission decisions for off-allowlist tools without bypass shortcuts; noGrantSurface denies permission-requiring operations promptly without raising an unanswerable card, and is valid only with enforcement. Safe rejection must be tested as rejection before effects, not accepted as a substitute for demonstrating actual enforcement in the two shipped backends.
Extend the shared suite with meaningful denied-mutation, restricted-turn and descriptor cases; first-party backends must pass through their real integration paths with keyless scripted runtimes. Fix the unknown-cost comment, align the public documentation and record this mandatory baseline in docs/decisions. Backend bugs exposed by the suite remain separately filed work, as scoped.
This resolves the decision. The issue has concrete acceptance criteria and no other needs label or known implementation blocker: remove needs: decision and add agent-ready. The published conformance requirements change; use contract/breaking labels, a CONTRACT: commit, same-commit contract documentation and a changeset naming the stricter requirements.
schlessera commentedon Sep 30, 2026
Claimed for the shared backend permission/posture and descriptor conformance suite, following the maintainer's mandatory-handling ruling. Claim token:
341-01a0f266. Intended branch:test/341-backend-conformance-01a0f266. I will exercise the first-party adapter paths with keyless scripted runtimes and record behavioral mutation receipts.schlessera commentedon Sep 30, 2026
Source correction (2026-09-30): refreshed the body’s moved source ranges against the explicit pre-fix baseline
dee9b10e. The defect description remains tied to that baseline so it will not incorrectly describe the tree after #661 merges. Scope and the maintainer’s mandatory-handling ruling are unchanged.schlessera commentedon Sep 30, 2026
Completed (2026-09-30): merged #661 as
c849bf27433d2af82ba4fb5e2ec20f3cb7b71aaband verified the squash commit is onorigin/main. All acceptance criteria are complete: observable denied tool bodies, mandatory restriction and safe-rejection cases, first-party actual enforcement, descriptor conformance, durable ruling and aligned public guidance.Verification: 179 focused tests; the full isolated local suite has 7,566 pass, 45 skip, 0 fail and 39,391 assertions; typecheck, lint and build pass. All 12 current-head PR checks pass, including browser and packaging gates. Six guard mutations failed on the expected behavioral assertions and were restored.
Out-of-scope findings are recorded separately: #659 for the existing unknown-cost lifecycle assertion (attached to #56), and #660 for doctor test isolation. No required action remains for #341.