Skip to content

fix(ai): take cumulative Anthropic stream usage from message_delta - #1074

Open
pcbeingused333 wants to merge 1 commit into
PostHog:mainfrom
pcbeingused333:fix/anthropic-stream-cumulative-delta-usage
Open

pcbeingused333 wants to merge 1 commit into
PostHog:mainfrom
pcbeingused333:fix/anthropic-stream-cumulative-delta-usage

Conversation

@pcbeingused333

Copy link
Copy Markdown

💡 Motivation and Context

Anthropic's message_delta usage is cumulative. With server tools (web search, web fetch, code execution), its input_tokens and cache counts grow past the message_start values during the response. The Anthropic stream accumulator only recorded input and cache counts from message_start (it reads output_tokens from message_delta), so streamed generations under-reported $ai_input_tokens and $ai_cache_creation_input_tokens.

Example from a recorded web-search stream: message_start has input 2247 and cache creation 0, while message_delta has input 2254 and cache creation 17269. Before this change the event reported 2247 and 0.

_AnthropicStreamAccumulator.consume, shared by the sync and async wrappers, now applies the message_delta input and cache counts after the per-event merge. It keeps the larger of the two values, since the totals never shrink, and leaves a field the event omits at the message_start value.

💚 How did you test it?

  • New test_streaming_takes_cumulative_message_delta_usage and its _async twin. They use real anthropic.types stream events with the values above and check the captured $ai_input_tokens, $ai_output_tokens and cache properties. Both fail on main.
  • posthog/test/ai: 951 passed, 15 skipped; on main, only the two new tests fail.
  • ruff format --check passes. ruff check reports the same findings as on main for the touched files. mypy … | mypy-baseline filter reports nothing new.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed. (n/a)
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file (added .sampo/changesets/cumulative-anthropic-stream-usage.md in the same format)

🤖 Agent context

Autonomy: Fully autonomous

  • The bug was found and the fix, tests and description were written by Claude Code; I reviewed the diff and the explanation before submitting.

  • Written with Claude Code (Claude Opus 5.5).

  • The fix keeps the larger of message_start and message_delta for each count, rather than switching the merge to "cumulative" mode. That mode would replace the merged raw_usage and lose message_start's fields, and max also tolerates a delta that omits or under-reports a field.

🤖 Generated with Claude Code

Anthropic's message_delta usage is cumulative. With server tools (web
search, web fetch, code execution) its input and cache counts exceed
message_start's, but the stream accumulator only recorded message_start's
input and cache counts, so streamed generations under-reported input
tokens and cache writes. Keep the larger of the two for each count.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@pcbeingused333
pcbeingused333 requested a review from a team as a code owner October 17, 2026 03:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant