Skip to content

math/aarch64/advsimd: Vectorize the powf special-case path - #99

Open
ptomsich wants to merge 1 commit into
ARM-software:masterfrom
ptomsich:ptomsich/powf
Open

ptomsich wants to merge 1 commit into
ARM-software:masterfrom
ptomsich:ptomsich/powf

Conversation

@ptomsich

Copy link
Copy Markdown

Lanes with x or y in {0, inf, nan} were fixed one at a time by a scalar copy of powf's special-case code, after the vector core had run on all four lanes, and an exact +0 also sent the vector through v_powf_x_is_neg_or_small. Exact zeros are common in clamped image data.

Replace the scalar fallback with a vector version of the same decision tree, and skip the core when every lane is special. When the only special lanes have non-negative x and finite nonzero y, and no other lane has negative or subnormal x, a new helper, v_powf_xspecial, fixes them inline. It computes 1 / (x * x) for negative y as an exponent-field flip, and special_case uses it too, so pow (+-0, y < 0) no longer raises FE_DIVBYZERO. Vector math does not guarantee exception flags. Special lanes no longer count as negative or subnormal, and the special-lane and x < 0x1p-126 tests share one branch and one vector-to-general transfer.

Neoverse N1 (built with GCC 16.1 at -O3), ns per element, y = 0.45:

  • clean inputs: 5.84 -> 5.77 (1.01x)
  • 25% negative lanes: 7.04 -> 7.23 (0.97x)
  • 50% zero lanes: 8.10 -> 6.94 (1.17x)
  • all zeros: 8.97 -> 1.52 (5.90x; scalar powf: 3.02)
  • all -0: 9.63 -> 2.55 (3.78x; scalar powf: 3.35)
  • one negative and one -0 lane per vector: 8.04 -> 8.77 (0.92x), as special_case now runs its odd-integer test in vector code where the old code used scalar code for the -0 lane.

Lanes with x or y in {0, inf, nan} were fixed one at a time by a scalar
copy of powf's special-case code, after the vector core had run on all
four lanes, and an exact +0 also sent the vector through
v_powf_x_is_neg_or_small. Exact zeros are common in clamped image data.

Replace the scalar fallback with a vector version of the same decision
tree, and skip the core when every lane is special. When the only
special lanes have non-negative x and finite nonzero y, and no other
lane has negative or subnormal x, a new helper, v_powf_xspecial, fixes
them inline. It computes 1 / (x * x) for negative y as an exponent-field
flip, and special_case uses it too, so pow (+-0, y < 0) no longer raises
FE_DIVBYZERO. Vector math does not guarantee exception flags. Special
lanes no longer count as negative or subnormal, and the special-lane and
x < 0x1p-126 tests share one branch and one vector-to-general transfer.

Neoverse N1 (built with GCC 16.1 at -O3), ns per element, y = 0.45:
- clean inputs: 5.84 -> 5.77 (1.01x)
- 25% negative lanes: 7.04 -> 7.23 (0.97x)
- 50% zero lanes: 8.10 -> 6.94 (1.17x)
- all zeros: 8.97 -> 1.52 (5.90x; scalar powf: 3.02)
- all -0: 9.63 -> 2.55 (3.78x; scalar powf: 3.35)
- one negative and one -0 lane per vector: 8.04 -> 8.77 (0.92x), as
  special_case now runs its odd-integer test in vector code where the
  old code used scalar code for the -0 lane.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution to Arm's optimized routines. Contribution to this repository does require a signed contributor's license agreement. See contributor-agreement.pdf for details

@blapie
blapie self-requested a review October 26, 2026 13:13
@blapie

blapie commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Thank you very much for this PR. This showcases some very good attempts at improving performance of special cases, that I think we should try to integrate, but I have additional concerns.

First, it looks like the current change is overly focused on cases that are considered rare in the design of AOR.
If such cases are common in the application we advise to branch directly in the user's code, I do understand that this may get in the way of autovec though.

Second, it is important that we leave the more general case of "clean" inputs as undisturbed as possible, as well as the common negative and subnormal path. At present I've also measured regression just above the noise level (~2%) on the faster paths (including positive, negative and subnormals), which could be acceptable, but also some more significant ones for negative or subnormals with inf/nan lanes (6%) and negative y (17%). What these number suggest is that it is hard to optimise for one set of special cases without damaging performance of other cases. And it is not obvious which case is most common in general.
My general feeling is that the gain is too small on cases with only 1 or 2 special lanes, to accept a regression in the faster paths and other special cases. A more detailed performance report would be useful to help the discussion.

Finally, a blocker IMO, but I'm open to discussion, is that the increased code complexity and size is too big for the overall benefit. There may be a way to share some of this logic with powr, which could mitigate this cost though.

There have been attempts in the past to optimize special cases but never successfully attained a satisfactory balance in overall performance (and in code complexity). The best compromise so far was to keep the simple scalar fallback, but I'm open to a vectorised solutions if it performs better overall (and some cases regress within reason).

@blapie

blapie commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

@ptomsich Is this optimisation intended to improve performance on a particular benchmark? If so which benchmark and how much gain do you observe?
Could the compiler be of any help to detect these cases better?

@ptomsich

Copy link
Copy Markdown
Author

@ptomsich Is this optimisation intended to improve performance on a particular benchmark? If so which benchmark and how much gain do you observe? Could the compiler be of any help to detect these cases better?

ocio in SPEC2026:

  • workload 4: 4.9%
  • overall benchmark: 2.4%

@TamarChristinaArm

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Arm's optimized routines. Contribution to this repository does require a signed contributor's license agreement. See contributor-agreement.pdf for details

This has been signed and filed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants