Skip to content

Add uniform usage/token telemetry across all AI providers #624

Description

@sroussey

Context

The AI provider run-fns (@workglow/ai + providers/*) do not surface token
usage today. Provider stream functions emit text-delta / object-delta and a
final finish event with {} (or the parsed object for json-mode); none of them
report input/output token counts, cached tokens, or reasoning tokens, and the
task layer has nowhere to read usage from.

This surfaced during the OpenAI Responses API migration: GPT-5.6 returns
usage.cached_tokens and usage.cache_write_tokens on the terminal
response.completed event (and every provider returns some usage), but there
is no channel to surface it, so the migration deliberately left telemetry out
rather than half-build an OpenAI-only path.

Goal

A uniform, provider-agnostic usage telemetry channel, wired across the board:

  • Anthropicusage.input_tokens / output_tokens / cache_creation_input_tokens / cache_read_input_tokens
  • OpenAI (Responses) — usage.input_tokens / output_tokens / input_tokens_details.cached_tokens / cache_write_tokens / output_tokens_details.reasoning_tokens
  • Google GeminiusageMetadata.promptTokenCount / candidatesTokenCount / cachedContentTokenCount
  • Ollama / llama.cpp / HFI — whatever each exposes (prompt/eval counts), normalized or left null

Design questions to settle

  1. Shape — a normalized Usage type (input, output, cached, cacheWrite,
    reasoning, total) with provider-specific extras under a loose bag? Where does
    it live (@workglow/ai)?
  2. Transport — how does usage travel from the provider run-fn to the task?
    Options: extend the finish event with an optional usage field (note: the
    streaming convention says finish.data is {} for text/tool — usage would be
    a sibling of data, not inside it, to avoid violating the no-accumulation
    rule), a dedicated usage stream event, or the run result metadata.
  3. Surfacing — logger (getLogger), task telemetry, and/or a field on the
    task output so workflows/agents and the builder UI can display cost.
  4. Multi-turn / agent loops — usage should aggregate across a tool-calling
    loop, not just the last request.

Scope

All AI providers, plus the @workglow/ai task/telemetry layer that consumes it.
Do it once, generically, so a new provider only has to fill in the normalized
Usage from its own finish payload.

Carved out of the OpenAI Responses migration
(prd/docs/superpowers/specs/2026-07-10-openai-responses-api-migration-design.md).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions