Skip to content

benchmarks: run_public_baseline.sh can't regenerate from a clean checkout (missing -static wrapper crates → 'Could not find libperry_runtime.a') #7012

Closed

Description

@proggeramlug

./benchmarks/run_public_baseline.sh cannot regenerate the public artifact from a clean checkout: every perry compile in the measurement legs dies with Could not find libperry_runtime.a.

Cause

The script builds only:

cargo build --release -p perry-runtime -p perry-stdlib -p perry

but perry-runtime / perry-stdlib are rlib-only — libperry_runtime.a and libperry_stdlib.a come from the separate perry-runtime-static / perry-stdlib-static wrapper crates. So target/release/ ends up with the perry binary but no static archives, and the compiler can't link anything it compiles.

This is invisible on a long-lived working copy (the maintainer's tree already has the archives from unrelated builds), which is why it has never been hit before. It reproduces reliably on a fresh git worktree / clone.

Why it's worth fixing rather than documenting

The failure lands at the worst possible moment. The script's shape is:

  1. cargo build --release …
  2. wait_for_quiet (needs CPU ≤25% for 60s, hard-aborts after 15 min)
  3. suite → wait_for_quiet → polyglot → wait_for_quiet → … (5 gates total)

So on a contended machine you wait hours for an idle window, the run finally starts, and then dies in the first measurement leg — and you wait for the next window to find out whether the fix worked. Total cost of the missing -static is measured in hours, not seconds.

Suggested fix

-cargo build --release -p perry-runtime -p perry-stdlib -p perry
+cargo build --release -p perry-runtime -p perry-stdlib -p perry \
+  -p perry-runtime-static -p perry-stdlib-static

Optionally add a fail-fast assertion after the build ([ -f target/release/libperry_runtime.a ] || fail "…") so a future packaging change surfaces immediately instead of during measurement.

Sequencing caveat for whoever takes this

run_public_baseline.sh is itself listed in HARNESS_PATHS, so editing it changes harness_fingerprint and invalidates any existing artifact (validate_public → "harness changed; regenerate"). The script fix and an artifact regeneration therefore have to land together — the fix cannot be shipped in the same PR as an artifact that was generated with the old script, and it can't be shipped alone without immediately reddening the lint freshness gate.

Found while regenerating the baseline to clear the lint gate after the workspace-restructuring merges (#6758/#6761) drifted source_fingerprint via Cargo.toml.

Activity

  1. proggeramlug commented on Jul 29, 2026

    @proggeramlug
    ContributorAuthor

    Second fragility found while working around this: HARNESS_PATHS includes the whole benchmarks/honest_bench directory, which contains npm lockfiles (workloads/4_http_fastify/node/package-lock.json). So a dependabot bump can silently invalidate the public artifact — #6752/#6753 changed 3 lines in that lockfile, which is enough to shift harness_fingerprint and make a freshly-generated artifact fail validate_public with "harness changed; regenerate".

    That turns baseline regeneration into a race against dependabot: the artifact has to be generated from a tree whose fingerprinted files match main at merge time, and a lockfile bump landing mid-run invalidates ~2 hours of measurement (5 quiet-CPU gates). It cost me one full window today — I had pinned a worktree, and the bump landed while I was waiting for an idle machine.

    Worth considering whether the honest_bench entry should fingerprint only the harness scripts + results/expected.json rather than the entire subtree, so vendored lockfiles don't participate. That is orthogonal to the -static fix above but hits the same workflow.

  2. added a commit that references this issue on Jul 30, 2026
    7f34165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions