Skip to content

Pin the ISA that source builds target in the base images - #50

Merged
rcannood merged 1 commit into
mainfrom
portable-native-builds
Aug 21, 2026
Merged

Pin the ISA that source builds target in the base images#50
rcannood merged 1 commit into
mainfrom
portable-native-builds

Conversation

@rcannood

Copy link
Copy Markdown
Member

Pin the instruction set that source builds target, instead of inheriting the build machine's.

hnswlib and annoy publish no wheels and their setup.py adds -march=native, so whether an image runs anywhere depends on which CPU the GitHub runner happened to be. An image built on an AVX-512 runner dies with Illegal instruction on anything older.

This is not hypothetical: in the last batch integration benchmark run, kbet_pg lost all 124 of its tasks to SIGILL while kbet_pg_label -- same source, same pegasus version, different runner -- passed all 248. import hnswlib alone was enough to crash. scanorama_correct lost all 6 the same way via annoy. Both images have since been rebuilt and are fine again, which is exactly the problem.

  • ANNOY_COMPILER_ARGS replaces annoy's whole flag list, so it repeats annoy's own defaults with -march=native swapped out.
  • HNSWLIB_NO_NATIVE only drops the flag, so CFLAGS/CXXFLAGS/CPPFLAGS are what put a defined ISA back. Three variables because build backends disagree about which one they read.
  • x86-64-v3 (AVX2/FMA/BMI2) is satisfied by every de.NBI node and every AWS Batch instance type we use. x86-64-v2 would be the fallback if anyone needs pre-2015 hardware.

Verified on base_python: hnswlib goes from 47 AVX-512 instructions to 0 while keeping its AVX kernels (323 ymm references, up from 316), annoy likewise, and both still import.

@rcannood
rcannood force-pushed the portable-native-builds branch from 67eb85e to 6cd865e Compare August 21, 2026 05:17
@rcannood
rcannood merged commit 39a29e0 into main Aug 21, 2026
5 checks passed
@rcannood
rcannood deleted the portable-native-builds branch August 21, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant