Skip to content

fix(ai): take cumulative usage and web searches from the Claude Agent SDK message_delta - #1112

Merged
marandaneto merged 1 commit into
PostHog:mainfrom
pcbeingused333:fix/claude-agent-sdk-message-delta-usage
Oct 23, 2026
Merged

marandaneto merged 1 commit into
PostHog:mainfrom
pcbeingused333:fix/claude-agent-sdk-message-delta-usage

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

In the Claude Agent SDK integration, _GenerationTracker takes the input and cache token counts from message_start and, from message_delta, only output_tokens. A message_delta usage is cumulative: with server tools (web search, web fetch, code execution) its input and cache counts grow past the message_start values, and it also carries server_tool_use.web_search_requests. Streamed generations therefore under-reported $ai_input_tokens and the cache counts, and $ai_web_search_count was never set (the Anthropic wrapper in this repo sets all of them since #1074).

The tracker now keeps the larger of the message_start and message_delta values for the input, cache read and cache creation counts (a field the delta omits keeps its message_start value), and sets $ai_web_search_count from server_tool_use.web_search_requests when it is positive.

💚 How did you test it?

  • test_message_delta_usage_is_cumulative_with_server_tools runs a stream whose message_start has 120 input tokens and whose message_delta has 5120 input, 300 output, 64 cache read, 32 cache creation and 2 web searches, and checks the captured $ai_generation properties. It fails on main (input 120, no cache or web search properties) and passes with the change.
  • test_message_delta_without_input_counts_keeps_message_start checks that a delta with only output_tokens leaves the message_start counts and does not add a web search count; it passes before and after.
  • posthog/test/ai/claude_agent_sdk: 44 passed.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed. (n/a)
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file (added .sampo/changesets/claude-agent-sdk-message-delta-usage.md in the same format)

🤖 Agent context

Autonomy: Fully autonomous

  • The bug was found and the fix, tests and description were written by Claude Code; the diff and the explanation were reviewed before submitting.

@pcbeingused333
pcbeingused333 requested a review from a team as a code owner October 22, 2026 22:26
… SDK message_delta

The tracker only read output_tokens from a message_delta. Its usage is cumulative,
and with server tools (web search, ...) the input and cache counts exceed the
message_start values, so streamed generations under-reported them, and the
number of web searches was never recorded. Keep the larger input and cache
counts, as the Anthropic wrapper does, and set $ai_web_search_count from
server_tool_use.web_search_requests.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@pcbeingused333
pcbeingused333 force-pushed the fix/claude-agent-sdk-message-delta-usage branch from 3a71630 to 48feaee Compare October 23, 2026 06:52

@marandaneto marandaneto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good; no qualifying findings. Verdict: correct.

@marandaneto
marandaneto enabled auto-merge (squash) October 23, 2026 12:31
@marandaneto
marandaneto merged commit 48889b6 into PostHog:main Oct 23, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants