Skip to content

fix(stack-protector): remove the check in every layout it is printed in - #903

Open
sjkim1127 wants to merge 2 commits into
mainfrom
fix/stack-protector-check-forms
Open

sjkim1127 wants to merge 2 commits into
mainfrom
fix/stack-protector-check-forms

Conversation

@sjkim1127

Copy link
Copy Markdown
Contributor

Stacked on #902 (its commit is the first one here). Merge #902 first; this PR then shows only its own commit.

What

-fstack-protector puts a check in front of every return of a function with a local buffer. stack_protector.rs removed it in one layout only — the handler call alone inside a one-armed if. The output still printed __stack_chk_fail in 44 -O0 and 1,071 -O2 functions of the full local DecBench configs. The checks among them have four layouts:

layout printed as
a label inside the guard (left by an earlier clone; unprinted when unreferenced) if (x) { __stack_chk_fail(); }
the handler after the check if (!x) { return v; } __stack_chk_fail();
the check jumps if (x) goto block_f;
the handler past the end of the arm the check is in if (c) { ...; if (!x) { return v; } } else { ... } __stack_chk_fail();

One rule covers them: a branch exactly one side of which does nothing but reach the handler is the compiler's check; it is replaced by its other side. A handler that control merely runs into — behind a call Fission does not know to be no-return — is not a check and stays.

Why this is better without mentioning a score: the reader is shown "if this value is non-zero, abort" in front of each return and has to recognise it as the canary. The source has no such decision.

Changes

  • stack_protector.rs: the rule above; after a removal, the handler block nothing reaches is swept (the handler only), and a jump left pointing at a removed label becomes the call it stood for.
  • cleanup/mod.rs: the check is removed before unwrap_else_after_diverging_then. The handler is a call that never returns; left in, it unwrapped an else the source wrote.
  • tail_dup.rs: reads the same definition, so the rule from fix(tail-dup): copy only a straight-line tail to more than one place #902 ("a return behind the check is still the function's return") holds for the handler-after layout.

Measurement

Against 0d3eb0cf2 (#902), full configs, same rows, caches off:

-O0 base -O0 change -O2 base -O2 change
total structure distance 392,704 392,625 619,683 616,056
structure perfect 12,994 12,994 7,379 7,409
Union 15,146 15,146 7,828 7,858
rows better / worse 26 (−184) / 15 (+105) 508 (−4,806) / 158 (+1,179)
new / lost perfect 0 / 0 30 / 0
functions printing the handler 44 3 1,071 529

Sample-set gate: perfect 67 → 67, total distance 7,611 → 7,592. (The gate prints REGRESSION 68 -> 67 against its stored floor; main does too.)

Every changed function printed the handler before; no other function changes. No output has a goto to a missing label, before or after.

What the numbers hide

  • 12 of the 30 new perfects are one function (tilde_expand_filename) in 12 openssh binaries.
  • The rows that got worse were read. In all 15 -O0 rows and 151 of the 158 -O2 rows the output already had fewer CFG nodes than the source, and removing the check's two nodes takes it further under: the check was standing in for nodes Fission fails to recover elsewhere. One row's +141 is the scorer switching from a size lower bound to a computed distance.
  • A first version re-ran the general unreachable-tail pass after removing a check. It deleted dead code that had nothing to do with the canary (list_disks: 371 lines → 13) and measured 1,593 better on -O2 than what is proposed here. Found by reading the largest line drops; replaced by a handler-only sweep.
  • Recompile-and-compare-bytes, on rows whose text changed: -O2 −3,385, -O0 +76. One -O2 function stopped compiling in the byte harness and three started; the cause of the one is not established.

Tests

15 new tests in stack_protector.rs, 1 in tail_dup.rs. Each rule was broken on its own (19 mutations) and the intended tests, and only those, failed; the table is in the proposal.

cargo nextest run -p fission-midend-structuring -p fission-pcode (debug and --release), the CI clippy line, cargo fmt --all --check.

Not fixed

  • 529 -O2 functions still print the handler because code runs into it: behind a call not known to be no-return (438), or behind a block one path of which runs out (79 — tail calls printed as __fission_branchind, switch without a default). Correct to keep until the code in front is known to diverge; that is no-return analysis, a different owner.
  • A handler nothing reaches where no check was recognised (11 -O2, 2 -O0).
  • The canary arithmetic stays as a dead assignment.
  • Not swept together with fix(normalize): a flag is a snapshot; SLEIGH's temporary flags are flags #900.

Proposal: docs/proposals/2026-11-16-stack-protector-check-forms.md

🤖 Generated with Claude Code

sjkim1127 and others added 2 commits November 16, 2026 02:43
tail_dup admitted any shared tail of at most six statements, including one
that contains an `if` or a loop, so a decision the program makes once was
printed at every place that reaches it: `if (!color) REMOVE_COLOR(); return`
four times in every BSD `RB_REMOVE`, a `do { } while (busy)` wait eight times
in libopencm3's `sdram_command`. angr's ReturnDuplicatorHigh, which the module
cites, duplicates only a return-only block.

Both admission points -- the gotoless split and the goto-driven clone -- now
require the region to be straight-line when it would end up in more than one
place. A single-reference region that is moved rather than copied is exempt.

The stack-protector check is the one branch that does not count: it is the
compiler's, it precedes every return of a protected function, and the last
cleanup pass removes it. `stack_protector_pass_arm` recognises it in the
two-armed shape it has when tail_dup runs; the existing predicate matches
only the settled one-armed guard. Without this exemption the same rule loses
25-27 perfects per config, all of them canary epilogues.

Measured on the full local DecBench configs against main, same rows:
unoptimized distance 394,148 -> 392,704, perfect 12,976 -> 12,994 (18 new,
0 lost); -O2 621,358 -> 619,683, perfect 7,378 -> 7,379 (1 new, 0 lost).
Sample-set gate: perfect 67 -> 67, distance 7,603 -> 7,611 (one row).
In the changed functions gotos rise (1,429 -> 1,695 and 1,732 -> 2,063) and
lines fall (39,192 -> 37,701 and 32,727 -> 31,052).

Proposal: docs/proposals/2026-11-16-tail-dup-straight-line-tails.md

Co-Authored-By: Claude Fable 5.2 <[email protected]>
The pass matched one layout: the handler call alone inside a one-armed
`if`. The check also reaches the output with a label in the arm (left by an
earlier clone, unprinted when unreferenced), with the handler laid out after
the `if` (`if (!x) { return v; } __stack_chk_fail();`), as a jump to the one
block that calls the handler, and with the handler past the end of the arm
the check sits in.

One rule now covers them: a branch exactly one side of which does nothing
but reach the handler is the check, and it is replaced by its other side.
A handler that control merely runs into (behind a call not known to be
no-return) is not a check and stays.

After a removal only the handler block nothing reaches is swept. Re-running
the general unreachable-tail pass there deleted dead code unrelated to the
canary (`list_disks`: 371 lines to 13) and scored better for it.

The check is removed before the `else` unwrap, not after: the handler is a
call that never returns and unwrapped an `else` the source wrote. `tail_dup`
reads the same definition of the check.

Measured against 0d3eb0c on the full local DecBench configs, same rows:
-O2 structure distance 619,683 -> 616,056, perfect 7,379 -> 7,409 (30 new,
0 lost); -O0 392,704 -> 392,625, perfect unchanged. Functions printing the
handler: 1,071 -> 529 and 44 -> 3. Sample-set gate: perfect 67 -> 67,
distance 7,611 -> 7,592.

Proposal: docs/proposals/2026-11-16-stack-protector-check-forms.md

Co-Authored-By: Claude Fable 5.2 <[email protected]>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 06:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants