Tune nearly all benchmarks to ~100M wasm instructions executed per iteration

benchmarks/README.md

@@ -69,39 +69,56 @@ record its execution, it must:
   and looking at the `dynamic_total_inst_count` column of the resulting CSV
   output.
 
+  Anything within 80,000,000-120,000,000 counts as on target. This matches
+  `TARGET_INST_COUNT` in [`scripts/pca.R`](../scripts/pca.R), which also drops
+  any benchmark measuring below `MIN_DYNAMIC_INST_COUNT` (half the target)
+  outright. Benchmarks that are far over the target dominate the wall-clock time
+  of a suite run; benchmarks under the floor disappear from the PCA entirely.
+
+* The knob that sets the size of the workload should be read from a sibling
+  input file at run time rather than hard-coded as a constant in the source, so
+  that the benchmark can be retuned without a Docker rebuild. Read it *before*
+  `bench_start()` so that the I/O is not measured, and keep a compiled-in
+  fallback for when the file is absent.
+
+  C and C++ benchmarks can use `bench_read_long()` from
+  [`include/sightglass.h`](../include/sightglass.h):
+
+  ```c
+  /* Fallback tuned so that this benchmark executes ~100M Wasm instructions. */
+  #define ITERATIONS 293
+
+  int iterations = (int) bench_read_long("./shootout-base64.iterations.input",
+                                         ITERATIONS);
+  bench_start();
+  for (int i = 0; i < iterations; i++) { ... }
+  bench_end();
+  ```
+
+  Name the file after the benchmark and the knob (e.g.
+  `shootout-base64.iterations.input`) when a directory holds several benchmarks,
+  or just `default.input` when it holds one. Keep the compiled-in fallback in
+  sync with the checked-in input file.
+
+  Two cautions when adding a knob:
+
+  * If the benchmark prints anything derived from the knob, its
+    `.stdout.expected` must be regenerated.
+
+  * Check that the compiler has not folded the workload away. Scaling a loop
+    down can let LLVM collapse it into a constant; `shootout-nestedloop`
+    executed *zero* instructions for exactly this reason. Verify the count
+    responds to the knob, and use the `BLACK_BOX()` macro from
+    `include/sightglass.h` (or a `volatile` local) to keep the work opaque.
+
 Many of the above requirements can be checked by running the `.wasm` file
 through the `validate` command:
 
 ```
 $ cargo run -- validate path/to/benchmark.wasm
 ```
 
-## Compatibility Requirements for Native Execution
-
-Sightglass can also measure the performance of a subset of benchmarks compiled
-to native code (i.e., not WebAssembly). To compile these benchmarks without
-changing their source code, this involves a delicate interface with the [native
-engine] with some additional requirements beyond the [Minimal Technical
-Requirements] noted above:
-
-[native engine]: ../engines/native
-[Minimal Technical Requirements]: #minimal-technical-requirements
-
-* Generate an ELF shared library linked to the [native engine] shared library to
-  provide definitions for `bench_start` and `bench_end`.
-
-* Rename the `main` function to `native_entry`. For C- and C++-based source this
-  can be done with a simple define directive passed to `cc` (e.g.,
-  `-Dmain=native_entry`).
-
-* Provide reproducible builds via a `Dockerfile.native` file (see
-  [`build-native.sh`](./build-native.sh)).
-
-Note that support for native execution is optional: adding a WebAssembly
-benchmark does not imply the need to support its native equivalent &mdash; CI
-will not fail if it is not included.
-
-## Additional Requirements
+## Additional Desiderata
 
 > Note: these requirements are lifted directly from the [the benchmarking
 > RFC][rfc].
@@ -119,11 +136,6 @@ following requirements:
 * A candidate program must be deterministic (modulo Wasm nondeterminism like
   `memory.grow` failure).
 
-* A candidate program must have two associated input workloads: one small and
-  one large. The small workload may be used by developers locally to get quick,
-  ballpark numbers for whether further investment in an optimization is worth
-  it, without waiting for the full, thorough benchmark suite to complete.
-
 * Inputs should be given through I/O and results reported through I/O. This
   ensures that the compiler cannot optimize the benchmark program away.
 
@@ -146,3 +158,50 @@ following requirements:
 
 * The corpus of candidates should include programs that use a variety of
   languages, compilers, and toolchains.
+
+## Benchmarks that cannot hit the ~100M instructions target
+
+A few benchmarks are deliberately out of band because the region they measure is
+a single indivisible operation whose smallest legal size still costs far more
+than ~100M instructions. Each is documented in its own `README.md`:
+
+| benchmark | instructions | why it cannot be reduced |
+| --- | --- | --- |
+| [`tract-onnx-image-classification`](./tract-onnx-image-classification/README.md) | ~6.6G | One MobileNetV2 forward pass; the model's input shape is fixed at 224x224. |
+| [`sqlite3`](./sqlite3/README.md) | ~459M | speedtest1's `szTest` is already at its minimum of 1. |
+| [`spidermonkey-markdown`](./spidermonkey/README.md) | ~289M | `marked`'s lazy inline-lexer regex compilation costs ~248M inside the measured region no matter how small the input is; `spidermonkey` is also in `build-all.sh`'s skip list, so only its input file can change. |
+| [`blind-sig`](./blind-sig/README.md) | ~245M | One RSA blind signature; the crate rejects moduli below 2048 bits. |
+
+Most of the `libsodium` subtests are also out of band, in both directions; see
+[`libsodium/README.md`](./libsodium/README.md). They are all built from one
+upstream test suite with a single per-test iteration count as the only knob, so
+a test whose body already costs more than 120M cannot be scaled down, and a few
+whose bodies are empty on Wasm cannot be scaled up.
+
+`noop` executes 0 instructions by design and is intended for measuring harness
+overhead.
+
+## Compatibility Requirements for Native Execution
+
+Sightglass can also measure the performance of a subset of benchmarks compiled
+to native code (i.e., not WebAssembly). To compile these benchmarks without
+changing their source code, this involves a delicate interface with the [native
+engine] with some additional requirements beyond the [Minimal Technical
+Requirements] noted above:
+
+[native engine]: ../engines/native
+[Minimal Technical Requirements]: #minimal-technical-requirements
+
+* Generate an ELF shared library linked to the [native engine] shared library to
+  provide definitions for `bench_start` and `bench_end`.
+
+* Rename the `main` function to `native_entry`. For C- and C++-based source this
+  can be done with a simple define directive passed to `cc` (e.g.,
+  `-Dmain=native_entry`).
+
+* Provide reproducible builds via a `Dockerfile.native` file (see
+  [`build-native.sh`](./build-native.sh)).
+
+Note that support for native execution is optional: adding a WebAssembly
+benchmark does not imply the need to support its native equivalent &mdash; CI
+will not fail if it is not included.

benchmarks/all.suite

@@ -79,15 +79,12 @@ libsodium/libsodium-secretbox7.wasm
 libsodium/libsodium-secretbox8.wasm
 libsodium/libsodium-secretbox_easy.wasm
 libsodium/libsodium-secretbox_easy2.wasm
-libsodium/libsodium-secretstream.wasm
 libsodium/libsodium-secretstream_xchacha20poly1305.wasm
 libsodium/libsodium-shorthash.wasm
 libsodium/libsodium-sign.wasm
 libsodium/libsodium-siphashx24.wasm
 libsodium/libsodium-sodium_core.wasm
 libsodium/libsodium-sodium_utils.wasm
-libsodium/libsodium-sodium_utils2.wasm
-libsodium/libsodium-sodium_utils3.wasm
 libsodium/libsodium-sodium_version.wasm
 libsodium/libsodium-stream.wasm
 libsodium/libsodium-stream2.wasm

benchmarks/blake3-simd/Dockerfile

@@ -1,21 +1,33 @@
 # This two-phase Dockerfile allows us to avoid re-downloading APT packages and wasi-sdk with every
 # build.
 
+ARG WASI_SDK_VERSION=28
+
 # First, retrieve wasi-sdk:
 
 FROM ubuntu:24.04 AS builder
+ARG WASI_SDK_VERSION
 WORKDIR /
 RUN apt update && apt install -y wget
 
-# Download and extract wasi-sdk.
-RUN wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-28/wasi-sdk-28.0-x86_64-linux.tar.gz
-RUN tar xvf wasi-sdk-28.0-x86_64-linux.tar.gz
+# Download and extract wasi-sdk for the host architecture, then normalize the install path to
+# `/wasi-sdk` so that the stage below does not have to know the version or architecture.
+RUN ARCH=$(uname -m); \
+    case "$ARCH" in \
+    x86_64) WASI_ARCH=x86_64 ;; \
+    aarch64 | arm64) WASI_ARCH=arm64 ;; \
+    *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; \
+    esac; \
+    TARBALL=wasi-sdk-${WASI_SDK_VERSION}.0-${WASI_ARCH}-linux.tar.gz; \
+    wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/$TARBALL; \
+    tar xvf $TARBALL; \
+    mv wasi-sdk-${WASI_SDK_VERSION}.0-${WASI_ARCH}-linux /wasi-sdk
 
 # Second, compile the benchmark to Wasm.
 
 FROM ubuntu:24.04
 WORKDIR /
-COPY --from=builder /wasi-sdk-28.0-x86_64-linux /wasi-sdk/
+COPY --from=builder /wasi-sdk /wasi-sdk/
 RUN apt update && apt install -y git patch
 
 # Set common env vars.

benchmarks/blake3-simd/README.md

@@ -1,6 +1,13 @@
 # BLAKE3
 
-This benchmark is similar to [../blake3-scalar] and should return the same hash result, but the
-build compiles BLAKE3's hand-written SSE2 implementation (`blake3_sse2.c`): `wasm_sse_compat.h`
-maps its x86 SSE2 intrinsics onto Wasm SIMD (via `<wasm_simd128.h>`), and a small patch forces
-BLAKE3's runtime dispatcher to select the SSE2 kernels on wasm.
+This benchmark is similar to [../blake3-scalar], but the build compiles BLAKE3's hand-written
+SSE2 implementation (`blake3_sse2.c`): `wasm_sse_compat.h` maps its x86 SSE2 intrinsics onto
+Wasm SIMD (via `<wasm_simd128.h>`), and a small patch forces BLAKE3's runtime dispatcher to
+select the SSE2 kernels on wasm.
+
+Both benchmarks hash whatever is in their own `default.input`, and both are sized to execute
+~100M Wasm instructions (see [../README.md]). Because the SIMD kernels do more work per
+instruction, this benchmark needs a larger input than [../blake3-scalar] to hit that target, so
+the two `default.input` files — and therefore the two expected hashes — differ. Feeding both
+benchmarks the same input does still produce the same hash; that is how
+`benchmark.stderr.expected` here was cross-checked.

benchmarks/blake3-simd/benchmark.stderr.expected

@@ -1,3 +1,3 @@
 [blake3] hashing ./default.input
-[blake3] input size = 3100000
-[blake3] returned 059bb0ac5450537c4d30544423687821a498c9af13c23782eed5e69fde7730ea
+[blake3] input size = 7220000
+[blake3] returned b1961ab611c4ed5f1ba0339a3b445c43ecc26ae1ae7926f93c351f0d28c6b631

benchmarks/blind-sig/README.md

@@ -5,3 +5,23 @@ crate](https://crates.io/crates/blind-rsa-signatures). Generating these
 signatures is a useful test of math on big integers.
 
 The `blind-rsa-signatures` crate is licensed under the MIT license.
+
+## Instruction count
+
+This benchmark executes ~245M Wasm instructions, which is above the ~100M that
+the rest of the corpus targets (see `benchmarks/README.md`).
+
+The measured region is a single blind signature — blind, blind-sign, finalize —
+so the only knob is the modulus size baked into `secret.der`, and private-key
+RSA work scales roughly cubically with it. `secret.der` is already at the
+smallest size the crate accepts: `blind-rsa-signatures` rejects any modulus
+outside 2048–4096 bits, and 2048 bits is what brought this benchmark down from
+~1.53G instructions at the original 4096 bits.
+
+Getting to ~100M would need a non-standard sub-2048-bit modulus, which the crate
+will not load, so this is left out of band deliberately. It is still well above
+`scripts/pca.R`'s `MIN_DYNAMIC_INST_COUNT`, so it continues to participate in
+the PCA.
+
+`benchmark.stdout.expected` is the signature itself, so it must be regenerated
+whenever `secret.der` changes.

benchmarks/blind-sig/benchmark.stdout.expected

@@ -1,16 +1,8 @@
-835F2D15A3D17F41D69F7CC87CE0DDCFFE36D8D7DF81621C60DF3E8749E50086
-9BFE20B7299F76DDAEEADCF400ABFAC5FA011D27AE63CD0F114F927D0FC9AA3E
-E21A59E78A588985E44B279E9C0C94221CCB612F5B8F34189E68B312FD29A5A5
-C6DF06A474EE406BDF3EC9615C7FD6C4F9791CFFC900F4CD926C1FE14AB6E977
-B25B785A180FD2D8EE69DC203258066FD8D20C977D88ECE58DFCACBDCA17F5FD
-9D0F34F61DDA5F4AF1D61161166B0643D8FC018FBB84A5E5B89EED284EB0C6B0
-154F99DF135D039CEFAF62CA397A02C8595432E8F43D18059B2C27B54B7D9CEC
-84281294576F8A8FF8A43A9CD97EAC5B2BDF2F18DCFC9B20CF09FCBEFB94821A
-9A0467A248523F6C4D33EFBAD114B648BD0B5BD1D256F95B1484E7522FAC0387
-35E8B2CDC7A85EF72CBF7697B649A891D89B88EB1F6DD1E8EE8C60BAD16A519D
-A429FD3055A3FAD5747B53B2AB669F20C88CBE0895FCAA69ACF6728C3471975C
-551E72B2087550196AFE50E0A99432A516D8C54FF45821E09D1C246738E59BC5
-5FC1A72BCA72A95CD66F75E83F7BF8112E53300213BAAA92257B9B28A06660E2
-3EBF69556A2156E1E54108D25329247D38DA77F7DE1948C6D90FC363C5EB88B4
-15B1DDED8F6369A4F0299EEBA7E8EAD4CDB52741FC660DCCF6724794E1675297
-CE48D3D4FFACB0BD3BF32AC5764D39D9CF9B8D1AF921F5244AB3F72AB44805F7
+372C5483F1D365CE9382DA0E1275E5FDA8E3B45BBE387DEE0E73B945294A9E0B
+37C5B16BA928EC2D491B104D02F78FAE8BEA776BFD7A068FE3B99F9C33091D82
+B61B8530271F47A1E09046CEA6873D503542FB85E8C3A62C736E76B6A9BBB7E8
+7C07009A5795A028C792A796602C41D095CB468A6FFC9987FDDEF1E924EEDCCD
+3CFF65713C3094C2176ADFAAB13DE24DE193D72B5A41E3CC8548518FD354DFB1
+CC3448354619BBCF0898B4658D5B8F078623AD9632798669134E701AFE4A3F1A
+9044B82311281BED0077A3EC1CB90EB0D50D9DB2404EFD1A1666AE655D1D3119
+5BA621FC9094A709FC81C0ACAD7D404759FDFB1A8D73ADE0D58E33A51D373F56

benchmarks/build.sh

@@ -30,7 +30,10 @@ print_header() {
 # To allow the use of symlinks in the benchmark directories (docker ignores them), we `tar` up the
 # directory and `--dereference` (i.e., follow) all symlinks provided.
 print_header "Create build context"
-TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX.tar)
+# Keep the `XXXXXX` at the end of the template: BSD `mktemp` (macOS) only substitutes *trailing*
+# `X`s, so a template like `...-XXXXXX.tar` is returned verbatim, and the next run then dies with
+# `mkstemp failed: File exists`.
+TMP_TAR=$(mktemp /tmp/sightglass-benchmark-dir-XXXXXX)
 # macOS's bsdtar bundles extended attributes (notably `com.apple.provenance`) that a Linux Docker
 # daemon rejects when unpacking the build context (`lsetxattr ... operation not supported`); exclude
 # them. GNU tar (used on CI) omits xattrs by default and lacks `--no-mac-metadata`, so only pass

benchmarks/gcc-loops/Dockerfile

@@ -1,21 +1,33 @@
 # This two-phase Dockerfile allows us to avoid re-downloading APT packages and wasi-sdk with every
 # build.
 
+ARG WASI_SDK_VERSION=28
+
 # First, retrieve wasi-sdk:
 
 FROM ubuntu:24.04 AS builder
+ARG WASI_SDK_VERSION
 WORKDIR /
 RUN apt update && apt install -y wget
 
-# Download and extract wasi-sdk.
-RUN wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-28/wasi-sdk-28.0-x86_64-linux.tar.gz
-RUN tar xvf wasi-sdk-28.0-x86_64-linux.tar.gz
+# Download and extract wasi-sdk for the host architecture, then normalize the install path to
+# `/wasi-sdk` so that the stage below does not have to know the version or architecture.
+RUN ARCH=$(uname -m); \
+    case "$ARCH" in \
+    x86_64) WASI_ARCH=x86_64 ;; \
+    aarch64 | arm64) WASI_ARCH=arm64 ;; \
+    *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; \
+    esac; \
+    TARBALL=wasi-sdk-${WASI_SDK_VERSION}.0-${WASI_ARCH}-linux.tar.gz; \
+    wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/$TARBALL; \
+    tar xvf $TARBALL; \
+    mv wasi-sdk-${WASI_SDK_VERSION}.0-${WASI_ARCH}-linux /wasi-sdk
 
 # Second, compile the benchmark to Wasm.
 
 FROM ubuntu:24.04
 WORKDIR /
-COPY --from=builder /wasi-sdk-28.0-x86_64-linux /wasi-sdk/
+COPY --from=builder /wasi-sdk /wasi-sdk/
 
 # Set common env vars.
 ENV CC=/wasi-sdk/bin/clang

benchmarks/gcc-loops/default.input

@@ -0,0 +1 @@
+191

benchmarks/gcc-loops/default.stdout.expected

@@ -1 +1 @@
-Results: (1e2fdc4e): ed7aa26a b9b28d5b 67808d19 b4412829 132bc4c6 132bc4c6 b4412829 eb3492a9 7e51915b 0 61ffda0c f95b0406 495fecb4 ab6b4a02 a1d16823 bdaa178a 0 67808d19
+Results: (efa423bd): ed7aa26a b9b28d5b 67808d19 b4412829 132bc4c6 132bc4c6 b4412829 eb3492a9 7e51915b 0 61ffda0c f95b0406 495fecb4 ab6b4a02 7345af92 bdaa178a 0 67808d19

benchmarks/gcc-loops/gcc-loops.cpp

@@ -346,7 +346,11 @@ int main(int argc,char* argv[]){
 #ifdef SMALL_PROBLEM_SIZE
   const int Mi = 1<<10;
 #else
-  const int Mi = 1<<18;
+  // The problem size is read from `./default.input` so that the workload can be retuned without
+  // recompiling; the fallback is tuned so that this benchmark executes ~100M Wasm instructions.
+  // `Mi` scales all of the kernels below uniformly, and the digests they print depend on it, so
+  // `default.stdout.expected` must be regenerated whenever it changes.
+  const int Mi = (int) bench_read_long("./default.input", 191);
 #endif
   init_memory(&ia[0], &ia[N]);
   init_memory(&ib[0], &ib[N]);

benchmarks/gcc-loops/sightglass.h

@@ -0,0 +1 @@
+../../include/sightglass.h