Skip to content

feat(decision): account for the decision layer's own tokens - #76

Merged
kemalelmizan merged 1 commit into
mainfrom
feat/decision-cost
Oct 17, 2026
Merged

kemalelmizan merged 1 commit into
mainfrom
feat/decision-cost

Conversation

@kemalelmizan

Copy link
Copy Markdown
Contributor

The gap

#74 made the planning pass's model calls visible in the run summary. The System One decision layer spends the same way and was still invisible:

  • With --system1-backend openai, every step makes a real, billed completion through /chat/completions. Nothing in the run reported its tokens.
  • Tool telemetry cannot see it: a decision is not a tool execution, and the decision layer is deliberately not on the ChatModel seam (src/decision/systemone.ts).
  • The usage the remote backend already parses (SystemOneResponse.usage) was dropped in SystemOneAdvisor.interpret.

So a run that leans on the remote decision tier understated its own spend, the same way an --plan-interval run did before #74.

The fix

Report every call. SystemOneAdvisor gains onCall, which fires for each backend call with the step, latency, the backend's tokens (when it reports any) and whether it answered:

this.onCall?.({ step: input.step, durationMs: Date.now() - started, ok: false });   // the backend threw
this.onCall?.({ step: input.step, durationMs: Date.now() - started, usage, ok: true });

It reports a call whose verdict was discarded (every answer below the confidence floor, so the loop never applies the decision) too — the tokens were spent either way. That mirrors PlanningPassOptions.onCall from #74.

Omit tokens rather than print zero. Both transports default the counts to 0 when a response carries no usage, so 0 means "not reported"; decision_tokens=0 would claim the decision was free. Same rule as plan_tokens.

Print it where a user looks. AgentRuntime accumulates the reports and exposes decisionCallSummary(), which the CLI appends to both run-summary sites:

[agent] stop=completed steps=4 plans=2 plan_tokens=1234 plan_ms=4100 decisions=2 decision_tokens=52 decision_ms=180

"" when the decision layer never ran, so a run without --system1 prints exactly what it printed before.

And in the REPL. /decision now ends with the same running total — the REPL has no run summary, and the decision layer is what that command is about:

confidence 0.80, model alpha, decoding deterministic, backend ollama:tev1:0.8b
  model=alpha (p=0.80); decoding=deterministic
  this run: decisions=2 decision_tokens=52 decision_ms=180

Notes

  • Tokens, not money. The remote tier's price lives with its provider, so the agent reports the count and does not invent a cost.
  • A shared concept is not built yet. There are now two accumulators: PlanUsage (a CLI-side host policy, feat(steps): report what the planning pass costs #74) and decisionCallSummary() (the runtime, because the decision layer is the runtime's seam). If a third instance of "the run's own model call" appears, that is the moment to look for one shape instead of three — the same reasoning that deferred ADR-004's coordinator.

Verified

  • SystemOneAdvisor: reports tokens on success (step included), reports a failed call with no tokens, reports a discarded verdict, and omits tokens a backend never reported.
  • AgentRuntime: decisionCallSummary() is "" before any call, decisions=1 decision_ms=… with a silent backend, and decisions=2 decision_tokens=52 decision_ms=… across a silent and a reporting call; lastDecisionSummary() shows the running total.
  • pnpm build, pnpm typecheck, pnpm lint, pnpm test:coverage — green: 335 passed / 5 skipped, coverage 90.4 / 84.0 / 89.4 / 90.4.

Pre-existing prettier drift in src/decision/advisor.ts, src/decision/advisor.test.ts and src/runtime.ts is left untouched (#49); the lines added here match prettier's output.

#74 made the planning pass's model calls visible in the run summary. The System
One decision layer spends the same way and was still invisible: with
`--system1-backend openai` every step makes a real, billed completion through
`/chat/completions`, and nothing in the run reported what it cost. Tool
telemetry never sees it either — a decision is not a tool execution, and the
decision layer is deliberately not on the `ChatModel` seam.

- `SystemOneAdvisor` gains `onCall`: every backend call, reported with the step,
  latency, the backend's tokens (when it reports any) and whether it answered.
  It fires for a call that **failed** and for one whose verdict was **discarded**
  (below the confidence floor, so the loop never applies it), because both were
  paid for — the same rule `PlanningPassOptions.onCall` follows.
- Tokens are omitted rather than reported as `0` when a backend reports no usage:
  both transports default the counts to `0` when a response has no `usage`, so
  `decision_tokens=0` would claim the decision was free.
- `AgentRuntime` accumulates what it reports and exposes `decisionCallSummary()`,
  which the CLI appends to both run-summary sites:

      [agent] stop=completed steps=4 plans=2 plan_tokens=1234 plans_ms=4100 decisions=2 decision_tokens=52 decision_ms=180

  `""` when the decision layer never ran, so a run without System One prints
  exactly what it printed before.
- `/decision` (the REPL's view of the decision layer, which has no run summary)
  now ends with the same running total, from the verdict outwards: the tier that
  answered, the rationale, then what the calls cost this run.

The remote tier's tokens are priced by its provider and invisible to this repo,
so the count is reported rather than converted to money — the agent does not know
the decision model's price. Noted in the docs.

Verified: `pnpm build`, `pnpm typecheck`, `pnpm lint`, `pnpm test:coverage` all
green (335 passed / 5 skipped; coverage 90.4 / 84.0 / 89.4 / 90.4). Tests cover
the report on success, on failure and on a discarded verdict, the omission of
tokens a backend does not report, the accumulation, and that `/decision` shows it.
Pre-existing prettier drift in `src/decision/advisor.ts`, `src/decision/advisor.test.ts`
and `src/runtime.ts` is left alone (#49); the lines added here match prettier's output.
@kemalelmizan
kemalelmizan merged commit b13f585 into main Oct 17, 2026
2 checks passed
@kemalelmizan
kemalelmizan deleted the feat/decision-cost branch October 17, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant