Skip to content

ci(pack): speed up ARM checks and restore Rust caches (pm) - #3450

Merged
fireairforce merged 4 commits into
nextfrom
zoomdong/ci-native-arm-cache-routing
Nov 10, 2026
Merged

fireairforce merged 4 commits into
nextfrom
zoomdong/ci-native-arm-cache-routing

Conversation

@fireairforce

@fireairforce fireairforce commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

Reduce CI time spent executing ARM snapshots through QEMU and rebuilding Rust dependencies, while retaining the existing platform matrices, product title gates, and regression tests.

  • Run Linux ARM GNU and musl checks on native ARM runners, with an ARM-native Alpine image for musl. Native musl tests omit Rust debug information and report resource usage to reduce and track linking overhead; test selection and debug assertions remain unchanged.
  • Restore target/profile-specific Rust caches. Cache Docker's actual Cargo and target directories; only the default branch writes shared caches, and cache failures do not block tests.
  • Skip Clippy and Web builds for changes limited to root README/agent documentation and docs/, using a small shared workflow with dorny/paths-filter. Unknown or unavailable change information keeps checks enabled. Clippy/Web no longer restart on PR body edits; workflows using product title gates retain their original triggers.
  • Preserve all six Pack targets, five PM targets, launcher checks, PM E2E platforms, Web E2E, and existing integration/benchmark scripts. Keep benchmark-label and manual dispatch behavior.

Measured CI results

First full validation at 508c1900a completed successfully in 30m30s. Ecosystem validation finished last. The measurements below predate the native musl test debug-information adjustment.

Compare previous successful Pack run with this PR's first Pack run.

Measurement Previous run This PR Observed improvement
Complete Pack workflow, including queue time 35m41s 26m11s 9m30s shorter (26.6%)
Linux ARM GNU job 35m07s 8m59s 74.4% shorter
Linux ARM musl job 30m06s 15m02s 50.1% shorter
ARM GNU: 167 snapshots 1122.60s 46.58s 24.1x faster
ARM musl: 167 snapshots 806.04s 51.58s 15.6x faster

All six Pack targets passed all 167 snapshots. In the same PR run, GNU x64 snapshots took 43.86s, close to native ARM GNU's 46.58s. The longest Pack job is now x64 musl at 23m30s; its compilation takes 19m07s and its snapshots take 72.88s.

Both compared runs used nightly-2026-11-06 and attempt 1. Repository revisions and runner allocations differ, so these are observed run comparisons rather than a controlled benchmark. The new Rust caches were misses; warm-cache savings remain unmeasured. Integration and performance jobs did not show a reduction in this cold run.

Test Plan

  • Verified the documentation filter excludes only the named root documents and docs/; nested Markdown snapshots, dotfiles, source changes, and unknown inputs still run checks. Verified missing/failed/skipped filter outputs conservatively run checks.
  • Passed actionlint workflow syntax and expression checks; no new ShellCheck diagnostics.
  • Passed cargo fmt --check, Tombi, Biome, typos, and git diff --check.
  • Compared platform targets and existing test commands against the base branch.
  • Native musl's full-debug test link was subsequently interrupted twice by runner shutdown (exit 143), without a Rust error or OOM diagnostic. The current test build disables only debug information and reports maximum RSS to validate a smaller link; the shutdown cause is not yet proven.
  • GitHub Actions passed: all six Pack targets, five PM targets, three npm-launcher platforms, four PM E2E platforms, Web WASM/browser E2E, Clippy, integration, and performance checks.
  • All eight ecosystem suites passed, including Ant Design, Umi on Linux/Windows, evjs, father, dumi, and ant-design-pro.
  • Current-revision validation is available in PR checks. The measurements above refer explicitly to the completed first validation.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T06:00:38.023593Z 508c190 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@fireairforce fireairforce changed the title ci(pack): speed up CI with native ARM and scoped caches ci(pack): speed up ARM checks and restore Rust caches (pm) Nov 10, 2026
@github-actions

Copy link
Copy Markdown

📊 Performance Benchmark Report (with-antd)

Utoopack Performance Report

Report ID: utoopack_performance_report_20261110_074801
Generated: 2026-11-10 07:48:01
Trace File: trace_antd.json (0.3GB, 0.87M spans)
Test Project: examples/with-antd


Executive Summary

Metric Value Assessment
Total Wall Time 5,945.1 ms Baseline
Total Thread Work (de-duped) 18,445.4 ms Non-overlapping busy time
Effective Parallelism 3.1x thread_work / wall_time
Working Threads 10 Threads with actual spans
Thread Utilization 31.0% ⚠️ Suboptimal
Total Spans 870,709 All B/E + X events
Meaningful Spans (>= 10us) 205,369 (23.6% of total)
Tracing Noise (< 10us) 665,340 (76.4% of total)

Build Phase Timeline

Shows when each build phase is active and how much CPU it consumes.
Self-Time is the time spent exclusively in that phase (excluding children).

Phase Spans Inclusive (ms) Self-Time (ms) Wall Range (ms)
Resolve 41,426 5,651.6 1,526.6 3,007.8
Parse 8,169 1,254.6 928.1 5,166.8
Analyze 136,152 40,104.6 7,901.0 5,104.2
Chunk 5,205 4,793.4 654.3 1,958.5
Codegen 11,706 1,997.9 1,270.6 1,953.4
Emit 30 40.7 20.5 8.0
Other 2,681 9,567.9 4,551.4 5,945.1

Workload Distribution by Diagnostic Tier

Category Spans Inclusive (ms) % Work Self-Time (ms) % Self
P0: Scheduling & Resolution 178,502 48,525.1 263.1% 9,923.3 53.8%
P1: I/O & Heavy Tasks 2,864 112.9 0.6% 92.6 0.5%
P2: Architecture (Locks/Memory) 0 0.0 0.0% 0.0 0.0%
P3: Asset Pipeline 22,914 7,995.5 43.3% 2,802.6 15.2%
P4: Bridge/Interop 0 0.0 0.0% 0.0 0.0%
Other 1,089 6,777.3 36.7% 4,033.9 21.9%

Top 20 Tasks by Self-Time

Self-time is the exclusive duration: time spent in the task itself, not in sub-tasks.
This is the most accurate indicator of where CPU cycles are actually spent.

Self (ms) Inclusive (ms) Count Avg Self (us) P95 Self (ms) Max Self (ms) % Work Task Name Top Caller
4,828.4 26,466.6 104,049 46.4 0.1 31.3 26.2% module module (64%)
2,719.0 2,950.6 21 129476.5 250.1 549.6 14.7% save snapshot persist (5%)
1,913.3 2,006.9 2,247 851.5 2.7 222.3 10.4% analyze ecmascript module module (78%)
1,133.6 11,595.2 29,824 38.0 0.1 6.1 6.1% process module process module (81%)
1,021.1 2,793.1 22,579 45.2 0.1 54.7 5.5% internal resolving internal resolving (73%)
864.6 1,191.2 5,993 144.3 0.5 65.5 4.7% parse ecmascript parse ecmascript (66%)
691.7 759.8 9,594 72.1 0.3 8.0 3.8% precompute code generation generate merged code (65%)
655.0 2,034.7 839 780.7 1.8 237.9 3.6% generate merged code chunking (81%)
546.1 4,594.9 4,118 132.6 0.2 37.2 3.0% chunking chunking (25%)
500.2 2,713.8 1,173 426.4 0.2 103.5 2.7% write all entrypoints to disk write all entrypoints to disk (10%)
496.8 2,849.9 18,189 27.3 0.0 3.9 2.7% resolving module (63%)
389.9 777.2 129 3022.6 4.4 236.2 2.1% emit code emit code (53%)
384.0 384.0 329 1167.2 1.0 228.6 2.1% generate source map code generation (83%)
242.8 242.8 14 17344.1 85.5 170.0 1.3% blocking save snapshot (43%)
194.9 854.1 1,783 109.3 0.2 52.1 1.1% code generation code generation (83%)
105.1 194.5 1,057 99.4 0.1 10.3 0.6% compute async chunks compute async chunks (47%)
63.4 63.4 2,174 29.2 0.0 2.6 0.3% read file parse ecmascript (91%)
20.1 20.1 11 1830.7 6.2 6.7 0.1% write file apply effects (100%)
17.3 76.8 419 41.2 0.1 2.5 0.1% async reference async reference (41%)
15.3 15.5 7 2185.9 10.4 14.5 0.1% create project None (0%)

Critical Path Analysis

The longest sequential dependency chains that determine wall-clock time.
Focus on reducing the depth of these chains to improve parallelism.

Rank Self-Time (ms) Depth Path
1 719.7 3 persist → save snapshot → blocking
2 476.3 6 chunking → generate merged code → emit code → emit code → emit code → read file
3 274.5 4 chunking → generate merged code → emit code → generate source map
4 256.3 2 save snapshot → blocking
5 222.4 3 process module → process module → analyze ecmascript module

Batching Candidates

High-volume tasks dominated by a single parent. If the parent can batch them,
it drastically reduces scheduler overhead.

Task Name Count Top Caller (Attribution) Avg Self P95 Self Total Self
process module 29,824 process module (81%) 38.0 us 0.06 ms 1,133.6 ms
internal resolving 22,579 internal resolving (73%) 45.2 us 0.08 ms 1,021.1 ms

Duration Distribution

Range Count Percentage
<10us 665,340 76.4%
10us-100us 129,067 14.8%
100us-1ms 65,825 7.6%
1ms-10ms 10,219 1.2%
10ms-100ms 216 0.0%
>100ms 42 0.0%

Action Items

  1. [P0] Focus on tasks with the highest Self-Time — these are where CPU cycles are actually spent.
  2. [P0] Use Batching Candidates to identify callers that should use try_join or reduce #[turbo_tasks::function] granularity.
  3. [P1] Check Build Phase Timeline for phases with disproportionate wall range vs. self-time (= serialization).
  4. [P1] Inspect P95 Self (ms) for heavy monolith tasks. Focus on long-tail outliers, not averages.
  5. [P1] Review Critical Paths — reducing the longest chain depth directly improves wall-clock time.
  6. [P2] If Thread Utilization < 60%, investigate scheduling gaps (lock contention or deep dependency chains).

Report generated by Utoopack Performance Analysis Agent

@fireairforce
fireairforce merged commit d0edcdd into next Nov 10, 2026
67 of 70 checks passed
@fireairforce
fireairforce deleted the zoomdong/ci-native-arm-cache-routing branch November 10, 2026 08:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants