[proggeramlug]

CENSUS-residual lever N4 ("cold paths that do not clobber"), cause g (spills). The census left the split of the stack traffic by cause unmeasured; this PR measures it first, and the result changes the fix: the hot path pays for the cold calls because LLVM is never told they are cold, not because of their calling convention. No convention changes.

Branch head: 9377102e3b. The branch merges current origin/main (b453cf8) by merge, not rebase.

Step 1: where the stack traffic comes from (measured)

Main (482d9c9), chess built for callgrind (PERRY_TARGET_CPU=znver3), exact per-instruction counts, tool /root/CHB/n4/s1/spill.py. Every instruction with a frame-relative memory operand is one access of a stack slot. From each access the tool walks the function's CFG to the neighbouring accesses of the same slot (backward from a load, forward from a store): two points where the value is in a register anyway. The calls on the way are what the value has to survive in between.

workload	total Ir	stack traffic, chess functions	only calls that run ≤ 2% as often as the access ("cold")	a call that runs ≥ 25% as often	in between	no call on the way
search 250000	33 603.9 M	2 644.1 M (7.87%)	1 296.5 M (3.86%)	1 075.2 M (3.20%)	43.7 M	227.4 M (0.68%)
perft 4	787.2 M	54.1 M (6.87%)	24.8 M (3.15%)	22.0 M (2.79%)	1.1 M	6.2 M (0.79%)
evaluate 100000	3 074.6 M	340.7 M (11.08%)	254.3 M (8.27%)	78.5 M (2.55%)	0	7.9 M (0.26%)
legal 10000	1 153.4 M	64.6 M (5.60%)	24.5 M (2.13%)	30.3 M (2.63%)	3.5 M	6.3 M (0.54%)

• The "cold" column is an upper bound for what a fix can remove: the allocator may keep some of these values in memory for other reasons.
• By callee, search (a value can be live across several): perry_array_read_cold_shared 956.0 M, js_gc_loop_safepoint 842.3 M, both and nothing else 669.4 M. Then cold sites of js_string_addref_if_heap_string 247.0 M, js_array_note_numeric_write_value 170.6 M, js_gc_note_slot_layout_aware 141.5 M, js_write_barrier_slot_validated_parent 97.1 M, js_array_record_enter_counted 86.7 M, js_dynamic_bitand 76.2 M. Evaluate: the cold read 230.1 M and the poll 227.6 M of its 254.3 M.
• Constants (cause b) are not stack traffic: they are rip-relative loads or operands from the NaN-box table. Not split here.
• GC roots (cause c): on every target with native roots (x86-64 and AArch64 Linux, Apple, x86-64 Windows) a root is an alloca ptr addrspace(1) promoted to SSA; rewrite-statepoints-for-gc relocates it at each collecting call, and the backend writes it to its stack-map slot right before the call and reloads it right after, inside the block of the call. For a cold call that is cold-block traffic (about 0 Ir). The explicit perry.native.home arrays are address-taken slots: 1.2 M in search.
• The calls that do run (the second column) are perry_array_index_normalize_i32 (6.7 M calls per site in make, unmake, evaluate), js_string_addref_if_heap_string at function entry, the push helpers, births and real callees. Not this lever.

Why the allocator spills on the hot path

No branch in the chess IR carried !prof, and js_gc_loop_safepoint and perry_array_read_cold_shared were not declared cold. Declaring them would not have been enough: rewrite-statepoints-for-gc runs before the optimizer, replaces a collecting call by gc.statepoint(..., @callee, ...), and keeps the attributes of the call site only (unit test the_statepoint_rewrite_keeps_only_the_call_site_attribute shows both halves). So LLVM's static estimate weighed every poll and cold arm like the code around it (icmp ne i32 %armed, 0 even reads as likely), and saving a value around them on their own path did not look cheaper than keeping it in memory.

What changes

• inprocess/cold_paths.rs, a module pass before the statepoint rewrite, for the helpers that exist only as the slow arm of an inline fast path (COLD_ARM_HELPERS in module/linkage.rs):
  • every call or invoke of one gets the cold call-site attribute, which the rewrite keeps on the statepoint;
  • a block whose every way on leads into a block with such a call is cold too; a conditional branch with exactly one cold side gets branch_weights 2000 : 1, the weights llvm.expect gives. A branch that already has weights keeps them; a branch with two cold sides gets none.
• The three helpers, also declared cold: js_gc_loop_safepoint (behind the arming word), perry_array_read_cold_shared (behind the plain-array guards), js_array_note_numeric_write_value (behind the F64-kind test; it clears the kind, so it runs once per array).
• The guard of the inline -, *, / and bitwise fast path (lower_guarded_numeric_arith) is wrapped in llvm.expect(…, true). Its other arm calls js_dynamic_*, which cannot be cold by name: the same helpers are the direct path of BigInt arithmetic. + is not touched: its slow arm is also string concatenation.
• Calls, callees, conventions, statepoints, relocations and stack maps are the same. The unit test counts the gc.relocate and gc-live of a marked and an unmarked function: equal.

Not marked, on purpose: the older COLD_HELPERS (the inline-cache misses and js_write_barrier_root_nanbox). Their declaration says cold, and under statepoints that reaches LLVM only where the call is not a statepoint. The first version of this pass marked every call of a cold callee. Measured on the experiment build (main 482d9c9), marking only those older helpers: TypeScript compiled from source +725 036 B of code (+1.13%) for 17 092.8 → 17 086.0 M instructions (−0.04%); three.js skin +37 547 B (+0.15%) for 5 460.3 → 5 455.6 M (−0.09%). That is size for nothing, so the pass names the slow-arm helpers instead. The list's comment says so.

Why no register-preserving convention

The same tool on this build, on vs PERRY_COLD_PATHS=0 (callgrind build, /root/CHB/n4/s-r3.txt):

workload	total Ir off → on	stack traffic off → on	of it, only cold calls on the way off → on
search 250000	33 610.6 → 32 442.6 M	2 644.1 → 1 755.2 M	1 296.5 → 451.1 M
perft 4	787.3 → 770.1 M	54.1 → 42.1 M	24.8 → 12.1 M
evaluate 100000	3 074.7 → 2 798.8 M	340.7 → 114.5 M	254.3 → 32.9 M
legal 10000	1 154.1 → 1 133.4 M	64.6 → 51.6 M	24.5 → 12.1 M

attacked goes from 400.5 M of stack traffic to 89.7 M in search, evaluate from 687.4 M to 237.9 M.

• Instructions. Of the 451.1 M left in search, the accesses for which the poll or the cold read is the only call on the way are 14.9 M (0.05% of search). The rest is live across helpers this PR does not mark (below) or is the slack of the upper bound. A convention under which the poll and the cold read clobber nothing could remove at most those 14.9 M.
• Size is where a convention would pay now. The save and restore that one spill used to share sit at each cold call: in attacked every poll and cold read has about ten instructions around its call (stores, reloads, a constant rematerialized). That is the size growth below. A callee that preserves the registers would delete those instructions.
• Not built. The thunk in front of each extern "C" entry has to save what the JS code's target features let the caller keep live (x86-64 preserve_all covers the vector registers of the caller's feature level, AArch64's covers a different set), and it puts a new kind of frame under every poll, which the collector's stack walk has to step over: the AArch64 walk follows the frame-pointer chain and decodes prologues, the x86-64 walk steps by unwind rows. That needs its own GC-stress pass on both targets.
• It is sound for the collector, for the record. Under native roots every managed pointer that is live across a collecting call is in its stack-map slot during the call, whatever the convention; a NaN-boxed double in a register is a copy that codegen re-reads from its root after the call (root_reload.rs). A preserved register is in the position a callee-saved register is in today.

Switch

PERRY_COLD_PATHS=0 (also off, false): no pass, the three helpers are declared as before, no llvm.expect on the arithmetic guard. It is a compile-cache input (compile_env.rs, test cold_path_switch_is_a_cache_input). The experiment build on main's 482d9c9 gave main's own chess counts with the switch off (33 147.9 / 772.6 / 3 095.1 / 1 130.9 M); the head's off arm is not compared against a main build at its merge base.

Results (qb9, instructions:u)

Build c4edbf06cc: this PR's code on main 71926f2. Every gate below ran on it. Main moved twice afterwards; the head merges it, and the last two rows of the gates table say what was checked after each merge. On vs PERRY_COLD_PATHS=0 on the same binary, two runs each, identical between runs, checksums equal in every run.

workload	on (M)	off (M)	change	census estimate for N4
search 250000	31 921.0	33 154.4	−3.72%	−1.5 to −2.5%
perft 4	751.7	772.7	−2.72%	−1.5 to −2%
evaluate 100000	2 810.7	3 095.2	−9.19%	−3 to −5%
legal 10000	1 110.8	1 131.5	−1.83%	−1.5%

The change is larger than the stack traffic it removes (search, callgrind build: −1 168 M instructions, −889 M stack traffic): the cold blocks also leave the loops, and the jumps around them go.

Mac (arm64, the ship target), /usr/bin/time -l instructions retired. Release build of c4edbf06cc, chess from ~/projects/chess, two runs each, checksums equal on and off:

workload	on (M)	off (M)	change
search 250000	31 843.9 / 31 837.2	32 760.8 / 32 759.7	−2.80%
perft 4	814.0 / 812.7	834.0 / 833.0	−2.40%
evaluate 100000	2 642.2 / 2 640.8	2 808.0 / 2 809.1	−5.90%
legal 10000	1 233.4 / 1 232.0	1 256.1 / 1 256.5	−1.81%

test_cold_paths.ts matches node there, on and off. No GC-stress run on the Mac (no instrumented runtime was built there).

What each part gives, from an experiment build with the cold set as a knob (qb9, main 482d9c9, weights 1 : 1 048 575, the older cold helpers marked too; off = 33 147.9 / 772.6 / 3 095.1 / 1 130.9 M):

arm	search	perft	evaluate	legal
poll + cold read, call-site cold only	32 356.2 (−2.39%)	762.2 (−1.35%)	2 848.9 (−7.95%)	1 121.8 (−0.80%)
poll + cold read, plus weights	32 140.9 (−3.04%)	758.6 (−1.81%)	2 813.7 (−9.09%)	1 118.7 (−1.08%)
+ js_array_note_numeric_write_value	32 137.0	759.2	2 810.1	1 118.9

• The weights matter on x86-64 and little on arm64 (same build on the Mac: search 31 953.5 → 31 935.5 M).
• The shipped weights are LLVM's llvm.expect weights (2000 : 1), not the worker-interrupt poll's 1 048 575 : 1.
• The llvm.expect on the arithmetic guard is the rest of the way to the table above: about −0.7% on search, −1.0% on perft and −0.8% on legal (the difference between the last experiment row and the head; it also contains the change of weights and of the base).

Gates

gate	result
stack traffic in the chess functions	search 2 644.1 → 1 755.2 M, perft 54.1 → 42.1 M, evaluate 340.7 → 114.5 M, legal 64.6 → 51.6 M (table above)
chess IR	on vs off: both modules differ (the helper declarations, the llvm.expect)
chess in check mode	perft, legal, search and evaluate rc 0, no misses
hello / hello2	byte-identical on/off (md5 dc07ba3f94af, 12f5e352baf4); callgrind Ir 127 145 and 623 929, the same in all 3 runs of each arm
parity fixtures, node v26.12.1	7 fixtures (the new control-flow/test_cold_paths.ts and 6 array fixtures) × on, off, check mode, PERRY_GC_MOVING_LOOP_POLLS=1, GC stress on, GC stress off, GC stress at the default rate: 49 of 49 MATCH
GC stress on the new fixture	PERRY_GC_SCHEDULE_SEED=7 PERRY_GC_SCHEDULE_RATE=1 (instrumented runtime, a moving minor at every handled safepoint): 664 forced copying minors, 9 860 moved objects, 84 070 loop polls taken; output equal to node, on and off
GC stress over the parity corpus	1 898 programs (array, class, control-flow, function, generator, iterator, object, map, set, string, destructuring, spread, error, async, promise), each compiled on and off with the instrumented runtime and run with seed 7 at the default rate and at rate 1. Compile rc and exit codes equal in every program and both runs. 12 compile in neither arm. Output differs on vs off in 2 programs, test_issue63_arr and test_issue63_escape, which print elapsed times. At rate 1, 1 051 programs reach a loop poll (the instrument exits 70 for the 834 that do not, in both arms); one runs into the 60 s timeout in both arms
parity IR census, on vs off	2 557 programs; no compile-rc difference; the saved IR differs in 2 554 (the declarations); 10 OUTDIFF: 5 programs that compile in neither arm, and test_issue63_arr, test_issue63_escape (elapsed times), test_date (the clock), test_math (Math.random), test_gap_this_param_receiver_routes (a pid in a warning)
vf, PERRY_VERIFY_FACTS=1	n=1538, json=0, abort=0, rc86=0, crc!=0=2 (the known pair: test_gap_s8b_computed_fields, test_gap_10554_fn_identity_own_module)
S0	rc 0, 71 executable controls pass
suites (release, qb9, c4edbf06cc)	perry-codegen 2850 passed, perry-wpfacts 2223 passed, num_seeds_audit 2 passed, perry-hir 881 passed, perry-runtime 6643 passed, 0 failed. perry compile_env: 5 passed, 1 failed: codegen_env_vars_are_build_cache_inputs ("shared list must be sorted and unique"), the failure main already has
-D warnings	perry-hir, perry-wpfacts, perry-codegen, perry: 0 errors
scripts/check_file_size.sh	passes on the gated commit. After the last merge of main it fails on crates/perry-stdlib/src/streams.rs (2448 lines against an asserted 2409), as it did on main 22ba61b itself; no file of this PR is involved
rustfmt	own file and own hunks only
Mac (arm64)	release build of c4edbf06cc; chess on/off above; test_cold_paths.ts MATCH on and off
after merging main 22ba61b (with PR-C3 #14402), build 98af3d327e	chess on / off: search 29 303.0 / 30 760.9 M (−4.74%), perft 685.6 / 710.5 (−3.50%), evaluate 2 517.0 / 2 847.7 (−11.61%), legal 1 048.8 / 1 073.4 (−2.29%), checksums equal, one run per arm; 3 fixtures × on, off, check mode: 9 of 9 MATCH; perry-codegen 2855 passed, 0 failed
the head	merges main b453cf8 (with the constant-array for-of, #14406) on top of that; git merge-tree clean; not built

PERRY_GC_STRESS, which earlier gate scripts of this lane set, does not exist in the tree: those stress arms were ordinary runs. The stress here is PERRY_GC_SCHEDULE_SEED with the instrumented archives (scripts/build_instr_runtime.sh), set at compile time and at run time.

Size (every executable section; on vs off)

.text, perry_js_text, .init, .plt, .plt.got, .fini. Every program grows.

program	on	off	change	run, instructions on / off
chess	7 621 385	7 607 817	+13 568 (+0.18%)	above
three.js skin	25 355 015	25 293 654	+61 361 (+0.24%)	5 446.9 / 5 460.3 M (−0.25%)
tssrc (TypeScript from source, transpileModule × 3)	64 156 691	63 951 897	+204 794 (+0.32%)	17 074.4 / 17 090.2 M (−0.09%)
Supabase CLI	218 120 413	217 714 205	+406 208 (+0.19%)	not run

The growth is the mechanism itself, not a side effect. Off, a value that is live across a poll or a cold read is spilled once and reloaded where it is used. On, it stays in its register on the likely path, and each cold call gets its own stores before and reloads after. attacked has 13 polls and 7 cold reads and grows by 1 093 B (to 6 250 B) while its stack traffic falls from 400.5 M to 89.7 M instructions. Per symbol: chess 38 functions grow, 1 shrinks by 4 B; tssrc 3 656 grow, 623 shrink; the largest single changes in skin, tssrc and Supabase are inlining shifts between a function and its specialized clone that cancel out (loadBufferView +9 183 against its $idx_u31 clone −9 230).

What was done to keep it small: the older cold helpers are not marked (that alone was +725 KB on tssrc, see above), and the weights are llvm.expect's. What would remove it: a convention under which the cold helpers preserve the caller's registers (above).

Programs that do not spend their time in polled loops over arrays pay the size and gain little: tssrc −0.09% instructions for +0.32% code. PERRY_COLD_PATHS=0 is the way back.

Not done, not covered

• No register-preserving convention (the section above): this PR removes the instructions, the convention would remove the bytes.
• Cold arms that stay unmarked, with what marking them gave on the experiment build (qb9, on top of poll + cold read, weights 1 : 1 048 575):
  • the store guards' barrier and layout-note arms (js_gc_note_slot_layout*, js_write_barrier_slot*, js_typed_feedback_array_set_*): search −0.9%, evaluate −0.8%. These helpers are the regular path of every pointer store, so they cannot be cold by name; each guard needs its own llvm.expect;
  • the array-record arms of for-of (js_array_record_*, js_rel_lt): search −0.8%, perft −1.1%. The constant-array for-of work removes them where the array is constant;
  • the generic arm of .length (js_value_length_property_f64): search −0.1%.
• The older COLD_HELPERS list is left as it is: declared cold, not seen as cold by LLVM where the call is a statepoint.
• The contribution of the llvm.expect alone is not isolated by a build of its own (the estimate above is a difference between two builds).
• Targets. Measured on x86-64 Linux and arm64 macOS. Nothing here is target-specific (an attribute, branch weights, an llvm.expect), and no other target was built. Shadow-stack targets get the same pass.
• Other programs' wall time was not measured on a quiet machine; the skin benchmark's own timer read 16.5 ms/pass on and 17.1–17.4 ms/pass off on the loaded box.
• The Mac GC-stress run (no instrumented runtime there), and a comparison of the head's off arm with a main build at the merge base.
• perry_array_read_cold_shared is not always cold: evaluate's own[file - 1] at file 0 takes it on 424 591 of about 7 M reads in search. The weights say 1 in 2000; the measured result with them is evaluate −9.19%.

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn