Skip to content

design: TDD 0067 rev 2 + ADR 0016 (escalation judged by the model that actually ran) - #186

Merged
cahenesy merged 1 commit into
masterfrom
docs/design/0067-actual-model-check
Oct 6, 2026
Merged

cahenesy merged 1 commit into
masterfrom
docs/design/0067-actual-model-check

Conversation

@cahenesy

@cahenesy cahenesy commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Resolves the 0067 BLOCKERS.md entry from /build-tdds run 20261005-174458.
In that run, the default escalation binding fable was refused headless
(credits_required), and an interactive Agent dispatch with model fable
silently ran claude-opus-5-5. All six fable-requested workers answered as
Opus. As revision 1 was designed, every escalation on this account would
have recorded escalated model=fable while Opus did the work (an NFR-4
violation).

  • TDD 0067 rev 2 (in-place revision; the TDD is still draft):
    • tl_worker_actual_model and tl_escalation_verify read the worker
      transcript (subagents/agent-.jsonl) after each escalated worker.
    • A family mismatch records fell-back "harness ran ". An unreadable
      model records fell-back "actual model unverified".
    • Substituted output is kept, and the TDD's later workers inherit.
    • The live probe now proves detection. P1 uses the binding, P2 a bogus
      model, and P3 is a positive control on opus that exercises the live
      transcript reader.
    • Build deviations folded in: credits_required counts as a fall-back;
      the async Agent stream shape is handled; the failed report is copied
      before retry-begin.
    • The diff sizes are re-declared to match the build.
  • ADR 0016 (new, accepted) revises only ADR 0015's fall-back detection
    consequence. Fall-back is now detected at dispatch OR after the worker
    completes, from its transcript. An unverifiable model counts as a
    fall-back, and a fall-back is never inferred from a timeout.
  • BLOCKERS.md: the 0067 entry is checked off with its resolution.

Recovery after merge: run /build-tdds, then step-3 Retry on the retained
branch build/20261005-174458/0067-build-tdds-escalation (c2d56fa). Do
not rebuild.

Design critique (design-reviewer, two fresh passes):

  • Pass 1: BLOCK.
    • M1: post-completion detection and unverified→fell-back contradicted
      ADR 0015's "detected at dispatch" and FR-88's wording, while the TDD
      said no ADR was needed. Fixed by ADR 0016, with the FR-88 wording
      listed under PRD conflicts surfaced as a candidate for a later PRD
      pass.
    • M2: the probe's PASS on this account never exercised the live
      transcript reader. Fixed by adding the P3 positive control.
    • Minors fixed: session-id scoping and validation, plus notes on Grok
      and unknown-family overrides.
  • Pass 2: DESIGN_REVIEW: PASS. Its two minors are applied: inherited
    workers are never verified, and an alias↔id family case is added to
    the fixtures.

Every TDD carries the reused, user-approved 0064–0067 evaluation rubric.

Open assumptions & waivers:

  • interactive fable dispatch silently substitutes opus — resolved: after each escalated worker, read subagents/agent-.jsonl message.model; a family mismatch → fell-back "harness ran ".
  • actual model may be unreadable — resolved: record fell-back "actual model unverified ()"; never claim an unproven escalation.
  • probe requires a usable binding — resolved: the probe proves detection. It passes iff each real dispatch is classified correctly. P3 is the positive control. BLOCKED only when the actual model can't be determined.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ki3TDnP2bNqqpEkFoZaEXq

…t actually ran)

Resolves the 0067 BLOCKERS.md entry from /build-tdds run 20261005-174458.
In that run, the default escalation binding `fable` was refused headless
(credits_required), and an interactive Agent dispatch with model `fable`
silently ran claude-opus-5-5. All six fable-requested workers answered as
Opus. As revision 1 was designed, every escalation on this account would
have recorded `escalated model=fable` while Opus did the work (an NFR-4
violation).

- TDD 0067 rev 2 (in-place revision; the TDD is still draft):
  - tl_worker_actual_model and tl_escalation_verify read the worker
    transcript (subagents/agent-<id>.jsonl) after each escalated worker.
  - A family mismatch records fell-back "harness ran <id>". An unreadable
    model records fell-back "actual model unverified".
  - Substituted output is kept, and the TDD's later workers inherit.
  - The live probe now proves detection. P1 uses the binding, P2 a bogus
    model, and P3 is a positive control on `opus` that exercises the live
    transcript reader.
  - Build deviations folded in: credits_required counts as a fall-back;
    the async Agent stream shape is handled; the failed report is copied
    before retry-begin.
  - The diff sizes are re-declared to match the build.
- ADR 0016 (new, accepted) revises only ADR 0015's fall-back detection
  consequence. Fall-back is now detected at dispatch OR after the worker
  completes, from its transcript. An unverifiable model counts as a
  fall-back, and a fall-back is never inferred from a timeout.
- BLOCKERS.md: the 0067 entry is checked off with its resolution.

Recovery after merge: run /build-tdds, then step-3 Retry on the retained
branch build/20261005-174458/0067-build-tdds-escalation (c2d56fa). Do
not rebuild.

Design critique (design-reviewer, two fresh passes):
- Pass 1: BLOCK.
  - M1: post-completion detection and unverified→fell-back contradicted
    ADR 0015's "detected at dispatch" and FR-88's wording, while the TDD
    said no ADR was needed. Fixed by ADR 0016, with the FR-88 wording
    listed under PRD conflicts surfaced as a candidate for a later PRD
    pass.
  - M2: the probe's PASS on this account never exercised the live
    transcript reader. Fixed by adding the P3 positive control.
  - Minors fixed: session-id scoping and validation, plus notes on Grok
    and unknown-family overrides.
- Pass 2: DESIGN_REVIEW: PASS. Its two minors are applied: inherited
  workers are never verified, and an alias↔id family case is added to
  the fixtures.

Every TDD carries the reused, user-approved 0064–0067 evaluation rubric.

Open assumptions & waivers:
- interactive fable dispatch silently substitutes opus — resolved: after each escalated worker, read subagents/agent-<id>.jsonl message.model; a family mismatch → fell-back "harness ran <id>".
- actual model may be unreadable — resolved: record fell-back "actual model unverified (<why>)"; never claim an unproven escalation.
- probe requires a usable binding — resolved: the probe proves detection. It passes iff each real dispatch is classified correctly. P3 is the positive control. BLOCKED only when the actual model can't be determined.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Ki3TDnP2bNqqpEkFoZaEXq
@cahenesy
cahenesy merged commit 8559bf6 into master Oct 6, 2026
1 check passed
@cahenesy
cahenesy deleted the docs/design/0067-actual-model-check branch October 6, 2026 01:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant