Collapse repeated per-request snapshot loads and provider-key resolutions in the chat dispatch path #941

Activity

1. coderabbitai commented on Aug 12, 2026

   

   coderabbitaiBot

   on Aug 12, 2026 – with coderabbitai

   More actions

   🔗 Related PRs

   #835 - feat(headers): request-context variables in default_headers + client-header forwarding allowlist [merged]
   #851 - perf(proxy): make request state clones constant-cost [merged]
   #852 - feat(obs): record client-abandoned requests as 499 [merged]
   #891 - feat(proxy): serve from thread-per-core workers with per-worker upstream pools [merged]
   #925 - perf(metrics): per-worker handle cache for every per-request emit path [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
2. coderabbitai

   added

   enhancementNew feature or requestNew feature or request

   P2Long-tail integrations — backlogLong-tail integrations — backlog

   on Aug 12, 2026
3. membphis

   closed this as completedin #953on Aug 12, 2026
4. membphis commented on Aug 12, 2026

   

   membphis

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Landed in #953 (squash-merged as a04a66c).

   What shipped. Both halves of this issue, across the whole client-facing
   handler family in one PR:

   • One Arc<AisixSnapshot> per request, threaded from handler entry through
     dispatch, the quota gate and every terminal emitter. A non-streaming chat
     request loads twice (the auth extractor, then the handler) instead of six
     times; the streaming path drops its request-start load as well.
   • One ProviderKey resolution per emitted attempt, via a new
     usage_attr::ResolvedPk that serves both the provider_key_name metric
     label and the usage event's telemetry_tags from a single row read —
     replacing three reads per request.

   request_metrics::Upstream now takes a PkLabels { id, name } pair rather
   than a bare id, which makes the collapse compile-forced: a call site cannot
   reintroduce a per-emit lookup, or pair a real id with a defaulted
   "unknown" name, without a compile error. That is why every endpoint —
   chat, messages (+count_tokens), responses, completions, embeddings,
   rerank, images, audio, videos, realtime, MCP, A2A, passthrough and the
   jobs surface, streaming and non-streaming — was converted in the same PR
   rather than left as follow-ups.

   Census (the primary evidence). Counting the distinct call sites
   SnapshotHandle::load is sampled under, on same-rig c=128 flamegraphs:

   call site	main (front)	main (back)	branch
   auth::authenticate_token	0.0388%	0.0570%	0.0748%
   chat::dispatch → chat::chat_completions	0.0323%	0.0618%	0.0326%
   request_metrics::record	0.0257%	0.0572%	gone
   chat::emit_usage_event	0.0194%	0.0333%	gone
   quota::reserve_policy_layers	0.0130%	0.0238%	gone
   request_metrics::record_usage	0.0064%	0.0143%	gone
   distinct call sites	6	6	2

   Those six are exactly the six this issue named, and both main legs
   reproduce the set independently. The ProviderKey resolution collapses the
   same way: two sampled call sites on main to one on the branch.

   Worth recording as method, because the first pass got it wrong: comparing
   frame shares between the two legs is not sound at this magnitude. The
   binaries are LTO'd independently and disagree about what is inlined, the
   folded captures carry different total sample counts (72.7G vs 99.9G), and
   at 0.01-0.07% per site the sampling error dominates — the unchanged
   auth load site alone swings +93% between the two main legs, and
   ResourceTable::get_by_name, which this change never touches, moved
   +135%. The call-site SET survives all of that, because an emit that no
   longer loads cannot appear as a parent of the load frame in any sample.

   Throughput: no regression, and the gain is not resolvable. Same-rig
   anchored A/B, main → branch → main, each leg its own build, full grid,
   25s x 4 windows, every window fail=0:

   leg	c=16	c=32	c=128	768@10ms	c=128 CPU/req
   main (front)	34,063	37,398	39,741	34,183	100.5 us
   branch	33,752	37,781	39,743	34,332	100.5 us
   main (back)	34,000	36,663	38,626	33,408	103.4 us

   The front-to-back anchor drift at c=128 is −2.81%, larger than the effect
   being measured. Against the front anchor the branch is +0.005%; against
   the linearly interpolated anchor pool (the branch leg ran at the session's
   temporal midpoint) it is +1.43%. The true value is somewhere in
   [0%, +1.4%] and this session cannot narrow it further. The 0.4-0.6 us this
   issue estimated sits inside that band — it was always likely to land below
   single-leg resolution, which is why the census was the acceptance
   criterion rather than the throughput delta.

   Two corrections to this issue's framing, both carried in the PR:

   • The "4-5 ProviderKey resolutions" figure counts the dispatch-side
     resolve_provider_key calls, which are routing's own work and out of
     scope here. The emit side did exactly three, and now does one.
   • The change surface is 20 files rather than "10+" — the emit-helper
     signatures reach the quota gate, the ensemble sub-call path and the
     crate's own tests as well as the handlers.

   Observable change, as this issue anticipated. A request's config view
   is frozen at handler entry, so an exporter added mid-request no longer
   reaches that request's usage event. Long-lived work is deliberately
   excluded: the end-of-stream emits and the realtime session-end emit read a
   fresh snapshot rather than pinning the request's, since either can outlive
   several config generations.

   Series names, label sets and label values are unchanged — verified by a
   main-vs-branch /metrics comparison over 24 driven cases spanning the
   whole family (including the single-attempt endpoints and the native
   Anthropic-protocol path), 859 identical series with no drift.