Repository navigation
feat(decision): account for the decision layer's own tokens - #76
Merged
Merged
Conversation
#74 made the planning pass's model calls visible in the run summary. The System One decision layer spends the same way and was still invisible: with `--system1-backend openai` every step makes a real, billed completion through `/chat/completions`, and nothing in the run reported what it cost. Tool telemetry never sees it either — a decision is not a tool execution, and the decision layer is deliberately not on the `ChatModel` seam. - `SystemOneAdvisor` gains `onCall`: every backend call, reported with the step, latency, the backend's tokens (when it reports any) and whether it answered. It fires for a call that **failed** and for one whose verdict was **discarded** (below the confidence floor, so the loop never applies it), because both were paid for — the same rule `PlanningPassOptions.onCall` follows. - Tokens are omitted rather than reported as `0` when a backend reports no usage: both transports default the counts to `0` when a response has no `usage`, so `decision_tokens=0` would claim the decision was free. - `AgentRuntime` accumulates what it reports and exposes `decisionCallSummary()`, which the CLI appends to both run-summary sites: [agent] stop=completed steps=4 plans=2 plan_tokens=1234 plans_ms=4100 decisions=2 decision_tokens=52 decision_ms=180 `""` when the decision layer never ran, so a run without System One prints exactly what it printed before. - `/decision` (the REPL's view of the decision layer, which has no run summary) now ends with the same running total, from the verdict outwards: the tier that answered, the rationale, then what the calls cost this run. The remote tier's tokens are priced by its provider and invisible to this repo, so the count is reported rather than converted to money — the agent does not know the decision model's price. Noted in the docs. Verified: `pnpm build`, `pnpm typecheck`, `pnpm lint`, `pnpm test:coverage` all green (335 passed / 5 skipped; coverage 90.4 / 84.0 / 89.4 / 90.4). Tests cover the report on success, on failure and on a discarded verdict, the omission of tokens a backend does not report, the accumulation, and that `/decision` shows it. Pre-existing prettier drift in `src/decision/advisor.ts`, `src/decision/advisor.test.ts` and `src/runtime.ts` is left alone (#49); the lines added here match prettier's output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
#74 made the planning pass's model calls visible in the run summary. The System One decision layer spends the same way and was still invisible:
--system1-backend openai, every step makes a real, billed completion through/chat/completions. Nothing in the run reported its tokens.ChatModelseam (src/decision/systemone.ts).usagethe remote backend already parses (SystemOneResponse.usage) was dropped inSystemOneAdvisor.interpret.So a run that leans on the remote decision tier understated its own spend, the same way an
--plan-intervalrun did before #74.The fix
Report every call.
SystemOneAdvisorgainsonCall, which fires for each backend call with the step, latency, the backend's tokens (when it reports any) and whether it answered:It reports a call whose verdict was discarded (every answer below the confidence floor, so the loop never applies the decision) too — the tokens were spent either way. That mirrors
PlanningPassOptions.onCallfrom #74.Omit tokens rather than print zero. Both transports default the counts to
0when a response carries nousage, so0means "not reported";decision_tokens=0would claim the decision was free. Same rule asplan_tokens.Print it where a user looks.
AgentRuntimeaccumulates the reports and exposesdecisionCallSummary(), which the CLI appends to both run-summary sites:""when the decision layer never ran, so a run without--system1prints exactly what it printed before.And in the REPL.
/decisionnow ends with the same running total — the REPL has no run summary, and the decision layer is what that command is about:Notes
PlanUsage(a CLI-side host policy, feat(steps): report what the planning pass costs #74) anddecisionCallSummary()(the runtime, because the decision layer is the runtime's seam). If a third instance of "the run's own model call" appears, that is the moment to look for one shape instead of three — the same reasoning that deferred ADR-004's coordinator.Verified
SystemOneAdvisor: reports tokens on success (step included), reports a failed call with no tokens, reports a discarded verdict, and omits tokens a backend never reported.AgentRuntime:decisionCallSummary()is""before any call,decisions=1 decision_ms=…with a silent backend, anddecisions=2 decision_tokens=52 decision_ms=…across a silent and a reporting call;lastDecisionSummary()shows the running total.pnpm build,pnpm typecheck,pnpm lint,pnpm test:coverage— green: 335 passed / 5 skipped, coverage 90.4 / 84.0 / 89.4 / 90.4.Pre-existing prettier drift in
src/decision/advisor.ts,src/decision/advisor.test.tsandsrc/runtime.tsis left untouched (#49); the lines added here match prettier's output.