feat(cache): wire prompt caching, the largest available cost reduction - #14
Merged
Merged
Conversation
EnablePromptCache has been a struct field that did nothing since Phase 0, documented as waiting on langchaingo to expose cache_control. That is no longer true: v0.1.14, already in go.mod, ships llms.WithCacheControl and its Anthropic adapter emits cache_control on human content blocks. The comment outlasted the constraint it described, and with it the biggest saving available to us. Measured against live rates on a 120k-token prompt whose first 100k are a stable prefix: glm-4.6 falls 66%, claude-sonnet-4-5 73%, gpt-4o 44% per warm hop. That is the shape of every vectorless tree navigation — a large document resent on each call, followed by a short question. Message.CacheBreakpoint marks the end of a cacheable prefix. Anthropic bills a cache read at a tenth of the input rate, so the saving is roughly proportional to how much of the prompt is stable. Two constraints are enforced rather than documented and hoped for. The marker only goes on user turns. langchaingo's system handler takes a bare TextContent and returns ErrInvalidContentType for anything else, so marking a system message would not cache it — it would fail every request. Assistant and tool turns are excluded for a different reason: a breakpoint belongs at the end of a stable prefix, and those turns are part of what varies. JSON mode appended its nudge by type-asserting the last part to TextContent. A cached part is not one, so the nudge would have been dropped silently and JSON mode would have become a no-op whenever caching was on. It now lands as its own part, outside the cached block, which is also where it belongs: the nudge carries a per-call schema, and changing the tail of a cached prefix invalidates it on every request. EnablePromptCache becomes a shorthand for the common shape, caching the first user message. It defers to the caller — an explicit breakpoint anywhere leaves the request untouched, since adding a second would cache a prefix nobody chose and providers cap how many a request may carry. The message slice is copied rather than marked in place, because Request.Messages belongs to the caller and a retry may hand the same backing array back. Cost accounting needed no change: providers meter caching in the usage they return, and HAL-529 already reads CacheWrite and CacheRead and prices them at their own rates.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
|
Warning Review limit reached
Next review available in: 2 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Reviewer's GuideThis PR wires provider-level prompt caching into the Anthropic adapter and core request/adapter flow, adds an explicit cache breakpoint API on messages, and ensures JSON mode and caller-owned message slices interact correctly with caching while being covered by targeted tests. Sequence diagram for Complete with prompt cache and JSON nudgesequenceDiagram
participant Caller
participant AnthropicConfig
participant AnthropicProvider as Anthropic.New
participant Adapter
participant AdapterHelpers as withPromptCache
participant Langchain as toLangchainMessages
Caller->>AnthropicConfig: set EnablePromptCache
AnthropicConfig->>AnthropicProvider: cfg
AnthropicProvider->>Adapter: NewAdapter(...)
AnthropicProvider->>Adapter: SetPromptCache(cfg.EnablePromptCache)
Caller->>Adapter: Complete(ctx, req)
Adapter->>AdapterHelpers: withPromptCache(req.Messages, promptCache)
AdapterHelpers-->>Adapter: msgsWithBreakpoint
Adapter->>Langchain: toLangchainMessages(msgsWithBreakpoint, req.JSONMode, req.JSONSchema)
Langchain->>Langchain: cacheable(TextContent, Message)
alt JSONMode
Langchain->>Langchain: append JSON nudge
end
Langchain-->>Adapter: []llms.MessageContent
Adapter-->>Caller: *llmgate.Response
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why now
EnablePromptCachehas been a struct field that does nothing since Phase 0, with this comment explaining it:That stopped being true. langchaingo v0.1.14 — already in
go.mod— shipsllms.WithCacheControl/llms.CachedContent, and its Anthropic adapter emitscache_controlon human content blocks (anthropicllm.go:327-353). The comment outlasted the constraint it described, and with it our biggest available saving. No new dependency, no native HTTP client, so this was never blocked on HAL-539.What it's worth
Measured against live rates, 120k-token prompt with a 100k stable prefix:
That's the shape of every vectorless tree navigation: a large document resent each call, then a short question.
API
Message.CacheBreakpointmarks the end of a cacheable prefix.anthropic.Config.EnablePromptCachestays as a shorthand that caches the first user message, for the common "big stable block first" case.Two constraints enforced, not just documented
The marker only goes on user turns. langchaingo's
handleSystemMessagetakes a bareTextContentand returnsErrInvalidContentTypefor anything else — so marking a system message wouldn't cache it, it would fail every request. Assistant and tool turns are excluded for a different reason: a breakpoint belongs at the end of a stable prefix, and those turns are what varies.JSON mode would have silently broken. It appended its nudge by type-asserting the last part to
TextContent; a cached part isn't one, so the nudge would have vanished and JSON mode become a no-op whenever caching was on. It now lands as its own part outside the cached block — which is also correct, since the nudge carries a per-call schema and changing the tail of a cached prefix invalidates it every request.The shorthand defers to the caller: an explicit breakpoint anywhere leaves the request untouched (a second breakpoint would cache a prefix nobody chose, and providers cap how many a request may carry). The slice is copied rather than marked in place, because
Request.Messagesbelongs to the caller and a retry may hand the same backing array back.Cost accounting
No change needed — providers meter caching in the usage they return, and HAL-529 already reads
CacheWrite/CacheReadand prices them at their own rates. So the saving shows up inResponse.Usage.CostUSDautomatically.Tests
Seven new cases in
internal/adapter/adapter_cache_test.gocovering the marker, the system/assistant/tool guard, shorthand placement, caller deference, the no-mutation guarantee, and the JSON-nudge interaction from both sides (nudge present, and outside the cached block).Not included
prompt_cache_min_tokens— caching below the provider minimum (1024 tokens on most Anthropic models) silently does nothing. Documented on the field; reading it from the feed is tracked in HAL-565.Closes HAL-541
Summary by Sourcery
Add configurable prompt caching support to reduce repeated prompt processing costs while preserving JSON-mode behavior and caller control.
New Features:
Enhancements:
Tests: