Repository navigation
perf(metrics): per-core histogram sample storage (sequenced after the allocator spike) #924
Closed as not planned
Description
Context
Split out of AISIX-Cloud#1259 item 3b. The worker-handle-cache PR removes the per-emit registration path; this issue tracks the OTHER metrics cost that cache cannot touch: histogram sample storage.
Every histogram record() pushes (f64, Instant) into the exporter's shared lock-free bucket. On the Graviton benchmark rig at c=128 saturation, that subtree costs ~1.98us/req across the four per-request histograms:
- block allocation (
calloc, 64-slot blocks, allocator arena contention across 4 TPC workers): ~1.19us - cross-core CAS on the shared bucket: ~0.43us
- epoch machinery: ~0.15us,
Instant::nowper sample: ~0.13us
A spike leg that diverted samples to a thread-local ring (same (f64, Instant) tuple, registration untouched) measured the upper bound: +1.1% throughput / 1.6-2.3us/req (rps vs drift-immune flamegraph basis).
Why not folded into the handle-cache PR
- Different mechanism and risk profile: a real per-core shard changes WHEN samples reach the exporter's storage. The unsolved design question is idle-worker flush — thread-local buffers are unreachable from the upkeep task, so either samples may sit buffered until the next request on that worker, or a cross-thread buffer registry is needed (shared state again, upkeep-only).
- Measurement sequencing: the largest share is allocator arena contention. The planned allocator spike (item 4 scope, todo 7) will shrink exactly that share; landing this first would double-claim the same microseconds. Measure this on a post-allocator baseline; if the marginal gain drops under the noise band, closing this as not-worth-it is a legitimate outcome.
Definition of done
- Design note answering the idle-flush question (bounded sample lateness vs buffer registry)
- Spike on the program rig with same-session anchors, c=128
- Go: implementation keeping series names/labels/values and summary semantics unchanged; No-go: numbers recorded here and closed
Reacted by coderabbitai
Activity
coderabbitai commentedon Aug 11, 2026
🔗 Related PRs
#798 - feat(guardrails): per-execution latency histogram
aisix_guardrail_latency_seconds[merged]#842 - feat(telemetry): split upstream and downstream latency on UsageEvent [merged]
#852 - feat(obs): record client-abandoned requests as 499 [merged]
#857 - perf(metrics): cache usage series handles [merged]
#870 - feat(obs): split the TTFT bucket set from e2e latency and make both configurable [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
perf(metrics): per-worker handle cache for every per-request emit path (
membphis commentedon Aug 11, 2026
Re-measured on the post-allocator baseline (main
b84d77b, jemalloc + fat LTO merged), same rig / same c=128 saturation methodology, 4/4 valid 25s windows, fail=0, 36.2K rps / 110.3µs CPU per request (matches the post-merge anchor within 0.3%).The histogram
record()subtree is now 0.40µs/req (0.367% of CPU), down from ~1.98µs pre-allocator:Instant::now: ~0.03µsFull-removal upper bound is therefore +0.37% throughput; a functionality-preserving thread-local ring realistically lands at +0.1–0.2%, below same-session anchor resolution (±0.07–1.0% drift) and far below the program's ±5% noise gate. The unresolved idle-worker flush design (bounded sample lateness vs a cross-thread buffer registry) would trade observable
/metricsfreshness semantics for a gain we cannot measure.Closing as not-worth-it per the definition of done. The remaining shared-bucket atomics stay on the contention-audit item's target list (#901 exporter scope), so that residual is still tracked.