Skip to content

ui-sdk: the AgentBackend contract suite never exercises a deny or the turn posture #341

Closed

Description

@schlessera

Context

Part of #56. Found by the 1.0 surface audit (#319), originally against main @ 550a41ef. The defect description and source ranges below are verified against the pre-fix baseline dee9b10e; they describe the behavior before PR #661.

#56's definition of done is a third-party provider that "still passes the
interface's contract test". For AgentBackend that test exists and is
published: runBackendContract in @schlessera/brain-ui-sdk/testing
(packages/ui-sdk/src/testing/index.ts:93). It covers capabilities shape,
profile listing, the turn lifecycle, abort, failure, per-session busy-ness and
unknown profiles (:100-397). It does not cover the permission surface, and
the permission surface is where the seam has moved since 0.33.0:

  • A deny is never exercised. packages/ui-sdk/README.md:48-50 says
    "permissions: true means a deny actually blocks a mutation, and the shared
    contract test suite enforces it". The suite's only permission code is a fake
    bridge that always allows (testing/index.ts:72). A backend that declares
    permissions: true and runs the tool anyway passes.
  • The turn posture has no conformance test and no declaration.
    StartTurnRequest.enforceAllowedTools (packages/ui-sdk/src/server/backend.ts:318,
    added in ecc93b9) and noGrantSurface (:342, added in 1de4d6c) each say
    "A backend that does not understand the field ignores it, which is why a
    [narrower] posture must also verify its backend honours it". No capability
    says a backend honours either, so a host cannot check. Both first-party
    backends are tested for it in their own suites
    (packages/ui-backend-claude/tests/no-grant-surface.test.ts,
    packages/ui-backend-pi/tests/grant-surface-posture.test.ts), which a
    third-party backend cannot run.
  • BackendModule has no conformance test. The descriptor is at
    packages/ui-sdk/src/server/backend-module.ts:153. Its context now accepts
    confirmBashPatterns: readonly ConfirmPatternSource[] | null (:100). The
    published suite has no descriptor profile parsing/resolution case a third-party
    author can run to detect incompatible assumptions.
  • costReporting contradicts the cost rule. BackendCapabilities.costReporting
    (backend.ts:64-65) says "result.costUsd is meaningful (0 otherwise)". The
    contract (docs/integration-contract.md:911-915, :1150-1151) says a cost is null or
    absent when unknown, never zero.

Scope

  • Add suite cases to runBackendContract: a backend with
    capabilities.permissions: true whose bridge denies a mutating tool does not
    run it, and emits the denial.
  • Maintainer ruling (2026-09-28, A): mandatory restriction handling by 1.0.
    Every conforming backend must honor enforceAllowedTools and noGrantSurface
    when requested, or reject an unsupported restricted turn before execution.
    Silent ignoring is forbidden. No optional capability flags or harness-only
    declaration. Optional request fields remain optional; ordinary turns retain
    their existing defaults. Update the misleading ignore-unknown-fields comments.
  • Extend the shared suite to exercise both restrictions and safe rejection.
    Enforcement means off-allowlist tools cannot bypass a permission decision;
    noGrantSurface denies promptly without an unanswered approval request, and
    remains invalid without enforcement. The two shipped backends must demonstrate
    actual enforcement, not pass solely by rejecting every restricted turn.
  • Record this baseline and rejected alternatives in docs/decisions, align the
    integration contract and authoring documentation, and explain that this is a
    pre-1.0 tightening of conformance, not an immediate general seam freeze.
  • Add a BackendModule descriptor case: profile parsing and resolution through
    defineBackendModule, run against both first-party descriptors.
  • Correct the costReporting doc comment to match the contract's cost rule.

Out of scope

Acceptance criteria

  • The maintainer selected mandatory handling (A), recorded 2026-09-28.
  • Source comments, integration/authoring docs and a durable decision record agree on mandatory handling, safe rejection and unchanged ordinary-turn defaults.
  • Both first-party backends pass the extended suite through their actual adapter paths with keyless scripted runtimes; denied tool bodies provably do not execute.
  • Off-allowlist operations require a permission decision; denied operations do not run. No-grant turns do not request an unanswered card; invalid posture combinations and unsupported restricted turns fail before execution. These cases are mandatory conformance coverage, not optional capability-gated tests.
  • Each new case fails when its guard is removed: a fake backend that ignores
    the deny, one that admits an off-allowlist tool under
    enforceAllowedTools, one that parks under noGrantSurface, and a
    descriptor that rejects a valid profile. The mutation and its failing
    assertion for each are recorded in the PR.
  • packages/ui-sdk/README.md's claim about the suite is true as written, or
    rewritten to say what the suite does.

Verification

bun run test; the mutation receipts in the PR.

Contract impact

  • This touches the integration contract, so it needs the contract label, a CONTRACT: commit prefix, and the doc updated in the same commit.
  • This changes a published package's behaviour, so it needs a changeset.

@schlessera/brain-ui-sdk/testing gains cases, so a third-party backend that
passed before may fail. That is the point of the issue, and the changeset says so.

Updated 2026-09-28: selected mandatory restricted-turn handling; refreshed source references, added durable record/documentation scope and concrete behavioral checks. Requires contract/breaking labels and a changeset naming stricter conformance requirements. Ready for implementation.

Updated 2026-09-30: verified and corrected source ranges against the explicit pre-fix baseline; PR #661 supplies the implementation and behavioral evidence. The original audit commit remains historical context.

Completed 2026-09-30 in #661, squash commit c849bf27433d2af82ba4fb5e2ec20f3cb7b71aab, verified on origin/main. The extended suite, descriptor cases, mandatory-handling record, aligned documentation and cost comment are delivered. Six restored behavioral mutations are recorded in the PR. Full isolated local suite: 7,566 pass, 45 skip, 0 fail; all 12 current-head PR checks pass. Separate findings #659 (unknown-cost lifecycle assertion) and #660 (doctor test isolation) remain open; neither is required unfinished work from this issue.

Activity

  1. added
    type: testCoverage, harness or proof work with no behaviour change.
    area: ui-sdkWire protocol, runtime schemas, the server and client SDK surfaces.
    area: backendsAgentBackend implementations and the seam they share.
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    securityAuth, isolation, egress, secrets, or anything in the threat model.
    on Sep 25, 2026
  2. schlessera commented on Sep 28, 2026

    @schlessera
    OwnerAuthor

    Decision (maintainer, 2026-09-28): Option A. Make handling of enforceAllowedTools and noGrantSurface mandatory for every conforming AgentBackend by 1.0, enforced by the published shared contract suite. Do not introduce optional posture capability flags or a harness-only opt-in. When a backend cannot uphold requested restrictions it must reject the turn safely before execution; it must never silently ignore them. These fields remain optional per-turn inputs; mandatory support does not turn the restrictions on for every ordinary chat turn.

    Preserve existing semantics: enforceAllowedTools requires permission decisions for off-allowlist tools without bypass shortcuts; noGrantSurface denies permission-requiring operations promptly without raising an unanswerable card, and is valid only with enforcement. Safe rejection must be tested as rejection before effects, not accepted as a substitute for demonstrating actual enforcement in the two shipped backends.

    Extend the shared suite with meaningful denied-mutation, restricted-turn and descriptor cases; first-party backends must pass through their real integration paths with keyless scripted runtimes. Fix the unknown-cost comment, align the public documentation and record this mandatory baseline in docs/decisions. Backend bugs exposed by the suite remain separately filed work, as scoped.

    This resolves the decision. The issue has concrete acceptance criteria and no other needs label or known implementation blocker: remove needs: decision and add agent-ready. The published conformance requirements change; use contract/breaking labels, a CONTRACT: commit, same-commit contract documentation and a changeset naming the stricter requirements.

  3. added
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    contractTouches the integration contract: CONTRACT: commit, contract doc updated in the same commit.
    breakingBreaks a published API or documented behaviour. Needs a maintainer ruling before code.
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    and removed
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Sep 28, 2026
  4. self-assigned this
    on Sep 30, 2026
  5. schlessera commented on Sep 30, 2026

    @schlessera
    OwnerAuthor

    Claimed for the shared backend permission/posture and descriptor conformance suite, following the maintainer's mandatory-handling ruling. Claim token: 341-01a0f266. Intended branch: test/341-backend-conformance-01a0f266. I will exercise the first-party adapter paths with keyless scripted runtimes and record behavioral mutation receipts.

  6. schlessera commented on Sep 30, 2026

    @schlessera
    OwnerAuthor

    Source correction (2026-09-30): refreshed the body’s moved source ranges against the explicit pre-fix baseline dee9b10e. The defect description remains tied to that baseline so it will not incorrectly describe the tree after #661 merges. Scope and the maintainer’s mandatory-handling ruling are unchanged.

  7. schlessera commented on Sep 30, 2026

    @schlessera
    OwnerAuthor

    Completed (2026-09-30): merged #661 as c849bf27433d2af82ba4fb5e2ec20f3cb7b71aab and verified the squash commit is on origin/main. All acceptance criteria are complete: observable denied tool bodies, mandatory restriction and safe-rejection cases, first-party actual enforcement, descriptor conformance, durable ruling and aligned public guidance.

    Verification: 179 focused tests; the full isolated local suite has 7,566 pass, 45 skip, 0 fail and 39,391 assertions; typecheck, lint and build pass. All 12 current-head PR checks pass, including browser and packaging gates. Six guard mutations failed on the expected behavioral assertions and were restored.

    Out-of-scope findings are recorded separately: #659 for the existing unknown-cost lifecycle assertion (attached to #56), and #660 for doctor test isolation. No required action remains for #341.

  8. added this to the 0.40.0 milestone on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.area: backendsAgentBackend implementations and the seam they share.area: ui-sdkWire protocol, runtime schemas, the server and client SDK surfaces.breakingBreaks a published API or documented behaviour. Needs a maintainer ruling before code.contractTouches the integration contract: CONTRACT: commit, contract doc updated in the same commit.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.securityAuth, isolation, egress, secrets, or anything in the threat model.type: testCoverage, harness or proof work with no behaviour change.

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions