perf(build): PGO (optionally + BOLT) for shipped release artifacts — x86 screening +31.5% / +36.6%, pending rig verification #967

Context

Split out of the perf program (AISIX-Cloud#1259). A local x86 screening pass (12-core VM; harness reuses the bench measurement core; clean-verified batch: first-to-last baseline drift −0.12%, floor headroom 11.9–16×, every run with full valid windows) measured, on top of the already-shipped allocator + fat-LTO release profile (#939):

Profile-dilution check: training on a three-dialect mix (chat + messages + responses) instead of the single measured endpoint cost only −1.09% — the gains live in the generic hot path (HTTP parsing / serde / dispatch), not in one endpoint's branches, so they are expected to transfer across traffic shapes.

Qualification that must travel with these numbers: measured against a synthetic small-body single-upstream workload. Per the platform decision below (2026-08-13), x86 is the primary verification platform for this work, so these numbers are the current gate-1 evidence; the remaining uncertainty is workload shape, not architecture. arm64 verification is deferred to a final acceptance pass.

Platform decision (2026-08-13)

x86 is the primary battlefield; the current x86 host environment is the verification rig. Most production deployments run on x86, so it is the more representative target. The arm64 (Graviton) benchmark rig is not a precondition for any early-phase work: early phases only need to demonstrate a clearly significant win on x86, and no arm regression runs are required until the end. A final acceptance pass on arm64 happens once the x86 side is complete. This supersedes the original "Graviton rig verification first" hard gate below.

Why this is a release-pipeline feature, not a code change

PGO/BOLT change no source. The deliverable is the build pipeline for shipped artifacts:

• PGO: build instrumented → run a training workload on the build host → rebuild with -Cprofile-use.
• BOLT (optional second stage): link with -Wl,--emit-relocs → llvm-bolt post-link layout optimization; needs the llvm-bolt toolchain on the builder.
• Channel analysis: the gains ship in our release tarballs and Docker images. Third parties building from source with a plain cargo build --release will not get them (committed-profdata schemes are toolchain-version-fragile; BOLT cannot ride a source build at all). This asymmetry is normal in the ecosystem — official binaries of major compilers, browsers and runtimes are PGO'd builds of the same source.
• Host-native constraint: the instrumented binary must execute on the build host, so cross-compiled targets cannot self-train. aarch64-linux artifacts need native arm64 runners (GitHub provides them). Survey note per the repo research rule: at least one mainstream Rust AI gateway ships mandatory fail-closed PGO for its host-native release targets while its cross-compiled aarch64-linux artifact remains un-PGO'd for exactly this reason; none of the mainstream gateways surveyed ship BOLT. Shipping PGO'd arm64 artifacts is therefore an open differentiation opportunity, and BOLT is one on every platform.

Training-set design (decided 2026-08-13)

Principles:

• A profile is a union of hotness, not a traffic-mix replica. Run each shape separately for a few thousand requests against a local mock, then merge the .profraw files (llvm-profdata merge). No "realistic concurrent user simulation" — determinism and reproducibility win.
• Coverage matters, proportions don't. Missing an entire hot path is the failure mode; a skewed ratio is not (bounded by the −1.09% dilution measurement above).

v1 shape matrix (~8 shapes):

• dialects: /v1/chat/completions, /v1/messages, /v1/responses
• streaming AND non-streaming — the throughput bench drives non-streaming only, but production streaming traffic must not be handed a cold layout
• body sizes: minimal (~86 B) and mid (~4 KB)
• config posture: default (observability on) for most shapes, plus one shape with a rate-limit policy configured so the quota path trains hot
• error paths: deliberately untrained — they are cold in production, and PGO treating them as cold is correct

Maintenance rule: the training mix mirrors the bench suites; when the product grows a new hot path, add a shape in the same PR.

Hard gates

1. x86 verification first (updated 2026-08-13; supersedes the original Graviton-first wording — see the platform decision above). Same-session anchored A/B on the x86 host per the program methodology (front/back anchors, fail=0, in-batch control leg, port-ownership pid preflight, floor headroom ≥2×). The clean 2026-08-13 screening batch (drift −0.12%, headroom 11.9–16×) already provides this evidence; re-run only if the training set or workload shape changes materially. arm64 (Graviton) verification moves to a final acceptance pass after the x86 work is complete.
2. Fail-closed. Once adopted, PGO is mandatory for release artifacts: any phase failing (instrumented build, training producing no .profraw, merge, optimized build) fails the release — never a silent fallback to a plain build. A positive proof marker (merged profile size + profraw count, written only after a successful -Cprofile-use build) is asserted by the release workflow before shipping.
3. Holdout validation. Keep one scenario out of the training set; the PGO'd binary must not regress beyond the noise band vs the non-PGO binary on that holdout. Re-run on every training-set change.
4. Training scripts and payloads are version-controlled; profiles are regenerated per release and never committed stale.

Measured costs (local, for planning)

• Build wall-clock: plain release 6m33s → PGO ≈16m (2.5×) → PGO+BOLT ≈23m (3.5×; BOLT itself is ~33 s — the rest is the full recompile that a RUSTFLAGS change triggers).
• The real CI requirement is running a workload mid-pipeline (once for PGO, once more for BOLT), not the toolchain dependencies.
• BOLT pitfall observed during screening: a distro /usr/bin/llvm-bolt resolves its runtime library by prefix and fails (library not found: /usr/lib/libbolt_rt_instr.a); the LLVM bin dir (e.g. /usr/lib/llvm-18/bin) must precede it on PATH.

Decision points (in order, all user-level)

1. x86 verification outcome (gate 1) — the clean screening batch stands as the current evidence.
2. Adopt PGO for release artifacts — accepting the CI complexity and ~2.5× release build time?
3. Adopt BOLT on top — ~+4% more for 3.5× build time and an extra toolchain, priced separately?
4. Interaction with the allocator candidate from the same screening pass (a combined allocator+PGO+BOLT stack cell was still being measured when this issue was filed; its result feeds here).

References

• rustc PGO guide: https://doc.rust-lang.org/rustc/profile-guided-optimization.html
• LLVM BOLT: https://github.com/llvm/llvm-project/tree/main/bolt
• Compile-time legs delivering 1:1 without shrinkage-law discounting (program precedent): perf(server): jemalloc global allocator + fat LTO release profile #939 and AISIX-Cloud#1259

Not scheduled — this issue is the self-contained handoff; implementation will be picked up by a dedicated session.