codegen: cold paths stay cold through the statepoint rewrite (N4)

changelog.d/codegen-cold-paths.md

@@ -0,0 +1 @@
+Performance: cold slow paths inside hot loops no longer cost the loop its registers. LLVM learns that a helper call is unlikely from the `cold` attribute of the helper's declaration, and uses it to place blocks and to decide where the register allocator saves a value that is live across the call. The GC statepoint rewrite, which runs before the optimizer, keeps only the attributes of the call site, so for every helper that can collect the fact was lost: the loop poll's call, the shared cold element read and the inline-cache misses were weighed like the code around them, and values live across them were stored and reloaded on the likely path. A pass before the rewrite now copies `cold` to each call site of a `cold` helper and gives a branch with exactly one side that leads only into such a call the weights `llvm.expect` would give. Three helpers that exist only as the slow arm of an inline fast path are now declared `cold`: `js_gc_loop_safepoint`, `perry_array_read_cold_shared` and `js_array_note_numeric_write_value`; and the guard of the inline `-`, `*`, `/` and bitwise fast path expects two Numbers. Calls, conventions, statepoints and GC roots are unchanged. `PERRY_COLD_PATHS=0` restores the previous code.

crates/perry-codegen/src/expr/bigint_bitwise_tests.rs

@@ -197,6 +197,25 @@ fn unproven_bitwise_operands_take_a_guarded_int32_arm() {
                 "{op:?} must mask its shift count to 5 bits:\n{ir}"
             );
         }
+        // N4: the guard says which arm is the likely one (`PERRY_COLD_PATHS`).
+        let guard = ir
+            .lines()
+            .find(|l| l.contains("label %guarded_arith.numeric"))
+            .unwrap_or_else(|| panic!("{op:?} has no guard branch:\n{ir}"));
+        let cond = guard
+            .split_whitespace()
+            .nth(2)
+            .unwrap()
+            .trim_end_matches(',');
+        let expected = ir.lines().any(|l| {
+            l.contains(&format!("{cond} = call i1 @llvm.expect.i1(i1 "))
+                && l.ends_with(", i1 true)")
+        });
+        assert_eq!(
+            expected,
+            crate::module::cold_paths_enabled(),
+            "{op:?} guard `{guard}` expects two Numbers exactly under the switch:\n{ir}"
+        );
     }
 }
 

crates/perry-codegen/src/expr/binary.rs

@@ -377,6 +377,13 @@ fn lower_guarded_numeric_arith(
         let fast_label = ctx.block_label(fast_idx);
         let slow_label = ctx.block_label(slow_idx);
         let merge_label = ctx.block_label(merge_idx);
+        // The helper is the arm for a BigInt or a value that needs ToNumeric.
+        let all_num = if crate::module::cold_paths_enabled() {
+            ctx.block()
+                .call(I1, "llvm.expect.i1", &[(I1, &all_num), (I1, "true")])
+        } else {
+            all_num
+        };
         ctx.block().cond_br(&all_num, &fast_label, &slow_label);
 
         ctx.current_block = fast_idx;

crates/perry-codegen/src/inprocess.rs

@@ -17,6 +17,7 @@
 
 mod assemble;
 pub(crate) use assemble::assemble_text;
+mod cold_paths;
 mod coff;
 #[cfg(test)]
 mod dead_statepoint_tests;