Repository navigation
Add profile-guided optimization (PGO) to improve mobile app startup performance #98008
Description
Proposal
Add profile-guided optimization (PGO) to Android and iOS release builds to improve mobile (app startup) performance, covering app-owned native code and relevant native dependencies built from source, including React Native, Hermes, JSI, Nitro, and Expensify native modules.
Background
Android and iOS compilers optimize native code by guessing which functions and branches run most frequently. PGO gives the compiler a recording of how the app was actually used — analogous to a traffic report — so it can make the busiest paths faster and spend less space on rarely-used ones. PGO optimizes compiled native code; it does not compile our JS into native code, but JS benefits indirectly because Hermes is native (faster Hermes init, bytecode loading, JSI communication, and native modules let NewDot become ready sooner).
Problem
Release builds currently use generic compiler heuristics with no knowledge of which native paths matter during startup and normal authenticated usage. Startup passes through many native components quickly: RN runtime init, Hermes VM startup and bytecode loading, JSI communication, Nitro/other native module init, native storage/networking/image/document integrations, and app-owned Kotlin/Java/Obj-C/Swift/C/C++ code. Some dependencies are consumed as prebuilt artifacts and cannot be further optimized with actual NewDot-usage info.
Solution
Add PGO to Android and iOS release builds. The representative workload focuses on cold startup of the HybridApp into NewDot. NewDot authenticated flows, OldDot, and hybrid transitions are not optimization targets (initially).
Benchmark results (directional local proof)
Android — faster phone (20 measured cold starts): Avg improved 4.75% (2951 → 2811 ms, −140 ms), P90 −166 ms (5.39%), P95 −185 ms (5.87%). Up to ~5–6% on mid-to-high-end devices.
Android — slower phone: Avg improved 1.85% (2721 → 2671 ms, −50 ms), P90 −45 ms, P95 −55 ms. Up to ~2% on low-end devices.
iOS — iPhone 14 Pro (10 measured cold starts): Avg improved 3.21% (1409 → 1364 ms), with P90 2.49% and P95 11.54%. Larger iOS gains likely because more code is compiled by the same toolchain than on Android.
How the PGO workflow works
- Identify a representative workload — cold authenticated startup plus (as the workload expands) opening/scrolling chats, sending messages, opening a workspace and changing reversible settings, uploading receipts/images/documents, viewing attachments, and search. Same logical journey on both platforms; OldDot/hybrid transitions excluded.
- Collect a profile — build with instrumentation counting native function/branch execution, run the journey on a physical device. Android writes
.profrawexplicitly (process is killed without exit callbacks), pulled viaadband merged withllvm-profdatafrom the same NDK. iOS uses Xcode to generate a profile on a physical device (not Simulator). Instrumented builds are never shipped. - Build using the profile — recompile app + relevant native deps in release mode with the profile so the compiler can inline hot small functions, arrange hot branches, co-locate hot machine-code blocks, and trade speed vs. size appropriately. Android and iOS share the journey but each produces its own profile artifact.
Implementation, profile lifecycle, CI, and safeguards
- Android tooling supports three modes:
release,instrumented,optimized. Relevant native dependency graph built from source; tooling verifies LLVM instrumentation and profile-writing APIs, pulls/merges per-library.profrawinto one.profdata, and archives APKs separately. - iOS will expose the same three logical modes; RN/Hermes/JSI/Nitro/app-owned native targets built from source with the right Xcode/Clang profile settings, mirroring the app-ready startup boundary. Finish the local iOS proof and collect equivalent numbers before enabling for production.
- Profile lifecycle & CI — initially manual; once validated, automate per release: build instrumented app → install on physical device → run representative journey → persist/merge profiles → build optimized release from same revision/toolchain → benchmark release vs. optimized → reject stale/incompatible profiles and surface warnings → publish optimized artifact only if perf and functional checks pass. Regenerate profiles every release; keep each profile tied to platform, source revision, build config, compiler/toolchain version, and architecture assumptions. Keep the perf marker/benchmark automation separate from the PGO implementation, and use the same measurement code for both builds.
- Risks & safeguards — workload must represent normal usage (avoid over-training one path); never ship or benchmark instrumented builds; native deps must be built from source; use compatible NDK/
llvm-profdata(Android) and Xcode/Clang (iOS); never share profiles across platforms; monitor APK/IPA size, build time, crash rate, and non-startup interactions; confirm OldDot/hybrid transitions still work via functional regression tests; record device model, OS version, thermal state, source revision, and build order, and counterbalance test order. If a release shows no meaningful improvement or a regression, the regular non-PGO release remains the fallback.
Follow-up ideas raised in discussion
- Measure impact on flows not covered by PGO (e.g. startup combined with the "last visited path" feature).
- iOS: combine PGO with order-file / temporal-instrumentation generation to further optimize main-binary layout for startup (temporal profile support for Swift landed in Xcode 26.4). Order files may also help on Android.
- Android complementary optimization: generate Baseline Profiles and Startup Profiles to precompile startup-critical Java bytecode and improve dex layout (with a workaround for prior HybridApp blockers).
PoC PR
Reported in Slack.
Activity
melvin-bot commentedon Aug 31, 2026
This issue has not been updated in over 15 days. @chrispader, @roryabraham eroding to Monthly issue.
P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!
roryabraham commentedon Sep 1, 2026
@chrispader how do you plan to prioritize this against the other work on your plate?
chrispader commentedon Sep 9, 2026
hey, sorry for the late reply! I'm finishing up my work on LegendList today, and then i can focus on PGO again. I can put focus on this for the rest of the week. Next week i will have limited availability (~50%/15-20h), as we have our Margelo workation going on.
chrispader commentedon Sep 23, 2026
@roryabraham i've continued working on #97808. I'm currently collecting final benchmark results for critical user flows and i'm working on integrating PGO optimization into CI, so we can apply it for production/staging builds
chrispader commentedon Oct 12, 2026
⚡ Update on the tooling for profiling, benchmarking and building PGO-optimized binaries for production ⚡
Following the Slack discussion, i've split the PGO work so reviewers can assess the tooling and benchmark reliability before we change which binary deployments ship.
The PRs for the evaluation phase should be reviewed and merged in this order:
@chrispader/perf/pgo-ci-evaluationinto Expensify/App and open the upstream PR so privileged physical-device jobs can run.All prerequisites must ultimately land upstream, but the foundation and scripts PRs can keep their Margelo source branches. The earlier CI prototype #31 and preview #102434 remain references and are outside this merge plan.Phase 1 is evaluation only. For each new native deployment candidate, CI will build an instrumented app, collect profiles on physical devices, build baseline and optimized binaries from the same source and toolchain, and compare them. A separate workflow runs Android and iOS independently and publishes results in its summary and artifacts, plus a candidate-specific comment on the deployment checklist with the workflow link.
Deployments continue with the baseline binary regardless of the evaluation result. Evaluation failures are reported without blocking deployment. Re-promoting the same binary reuses its report, and manual reruns remain available.
Phase 2 enables conditional rollout through a separate PR. After Chris and Expensify engineers review repeated results across multiple deployments, we can manually enable automatic selection independently for Android and iOS. We will agree on numerical thresholds based on the variability measured in phase 1.
An optimized candidate must show a repeatable, meaningful improvement in at least one agreed primary benchmark, with no meaningful regressions elsewhere and passing correctness checks. Failed profiling, invalid artifacts, failed benchmarks, or inconclusive results keep the baseline. Selection happens before staging upload, so production promotes the same binary that passed staging QA.
Phase 3 would make optimized builds the default. Once multiple optimized releases have shipped successfully, a later PR could remove the comparison from every deployment. We would retain periodic comparisons and additional comparisons after significant React Native, Hermes, or toolchain changes, alongside profile and correctness checks. This phase is future work.
Final recorded performance improvements
The latest automated benchmarks compared the mean latency of baseline and PGO-optimized builds on physical devices. Positive reductions below mean lower latency.
Android, Samsung SM-G970F, October 8
Each comparison contains 20 measured samples per binary and metric in each build order. Incomplete batches were excluded.
Each benchmark has 40 measured samples per binary across both build orders. Warm-ups and incomplete batches are excluded.
The Android builds had FullStory build processing disabled, so they are not store-equivalent.
iOS, iPhone 13 Pro, September 23–25
Startup and navigation comparisons contain 20 measured samples per binary. The heavy-report comparisons contain five samples per binary in each build order.
The heavy-report metric can finish at a skeleton. It does not establish complete report readiness or smoother scrolling.
Next, we need to review the prerequisite PRs, set up the CI workflows and test them in PR 3. Device coverage should include a slower and a faster physical device per platform where infrastructure supports it. Local benchmarks and the recent physical-iPhone pipeline smoke are useful evidence, but actual CI execution and store export remain pending.
cc @roryabraham @Julesssss
chrispader commentedon Oct 13, 2026
@MelvinBot please assign and request review from @Julesssss and @roryabraham on the PRs mentioned above.
MelvinBot commentedon Oct 13, 2026
I assigned
Julesssssandroryabrahamto [Task 1/5] [No QA] Add native PGO build and benchmark tooling and requested their review.I couldn't update the other three PRs:
margelo/Expensifyfork.You can add them yourself on those PRs. You can also add them once the Margelo branches have upstream PRs in Expensify/App.
view run