Skip to content

Run the tooling tests and the suites under a second thread environment - #11

Merged
MuteJester merged 3 commits into
mainfrom
ci/tooling-tests-and-thread-envs
Nov 15, 2026
Merged

MuteJester merged 3 commits into
mainfrom
ci/tooling-tests-and-thread-envs

Conversation

@MuteJester

Copy link
Copy Markdown
Owner

Two checks this repository owns ran in no workflow. Both had the shape the
verification gate had before #8: a check nobody runs does not report a
regression late, it reports nothing.

verification/tests now runs in CI

114 tests guard the gate itself — the comparison logic, the record selector,
the inventory readers, the GIL-region and thread-count scans, the move
bijection. They ran by hand only. Two of them are hardcoded tripwires (the
ALL set in test_record_selector, the Makefile target spelling in
test_wrap_link_guard), and a tripwire nobody runs is a comment.

They stay outside tests/ deliberately: CI copies tests/ elsewhere and runs
it against the installed package, so a test that imports this repository's
tooling cannot live under it. The new Verification tooling job builds the C
binaries first, so the three test_build_dependencies cases that decline to
judge a stale build actually run.

The suites now run under more than one thread environment

Every CI run of the C suite has used the runner's default of two threads, and
the six Python legs run with OMP_NUM_THREADS unset — which
tests/conftest.py already records as the condition that lets every test
claim every core and makes several thrash rather than run.

Two real defects in the parallelism work appeared only under a non-default
cap: an assertion whose two outcomes collided on the same number, and a policy
clause reading omp_get_num_procs() where it needed omp_get_max_threads(),
which planned sixteen threads under OMP_NUM_THREADS=2. One environment
cannot find that class.

Thread environments runs the C suite at 1 and at 3, and the Python suite at
3. One thread is where a reduction or a per-thread buffer can be skipped
entirely. Three is odd, divides no chunk count evenly, and — unlike another
power of two — cannot collide with the plan fallback that once made that test
pass by luck.

At one thread the C suite reports 438 passed and 1 skipped: test_plan.c:607
needs two threads to tell "the worker inherited the plan" from "the worker fell
back", and skipping is correct there.

What the new tests prove, and what they do not

They assert the property, not the YAML spelling, and each was mutation-tested
against the real workflow: commenting out the tooling step, moving
--threads 1 to 2, moving --threads 3 to 4, and dropping the
OMP_NUM_THREADS pin each fail exactly one test and leave the other six
passing. A commented-out job does not count as present.

They prove a command is present outside a comment. They do not prove the
job is reached, that its if: lets it run, or that it passed — the same limit
the thread-plan wiring line turned out to have. That limit is stated in the
test file rather than left to be discovered.

Verified locally

  • C suite at 1 thread: 439 tests, 438 passed, 1 skipped, 0 failed
  • C suite at 3 threads: 439 tests, 439 passed, 0 failed
  • Tooling suite: 114 passed, 3 skipped
  • Gate: exit 0, 14 OK lines, no FAIL and no notes — every recorded number
    byte-identical, including the 78 numeric keys, the 58 bit-identical math
    values and the 416 exact quantities
  • Workflow YAML parses; both new jobs well-formed

No C or Python source changed, so no baseline moved and the extension is
untouched.

Also

The gate grew a thread counts line in 7f3717e and the table in
verification/README.md never gained the row — the one check that catches a
region running at one thread per core regardless of the plan was undocumented
in the file that explains the gate. Added, along with two counts that had
drifted with the suites they describe (the C half is 439 tests, not 331;
exact holds 416 quantities, not 351). Neither number is recorded anywhere the
gate reads, which is why they could drift.

Two of them ran in no workflow, and both had the shape the verification
gate had before 845a4f2: a check nobody runs does not report a regression
late, it reports nothing.

`verification/tests` is the suite guarding the gate itself -- the
comparison logic, the record selector, the inventory readers, the
GIL-region and thread-count scans, the move bijection. 114 tests, run by
hand only. Two of them are hardcoded tripwires, the `ALL` set in
test_record_selector and the Makefile target spelling in
test_wrap_link_guard, and a tripwire nobody runs is a comment.

The other is the thread environment. Every CI run of the C suite has used
the runner's default of two threads, and the Python legs run with
OMP_NUM_THREADS unset, which tests/conftest.py already records as the
condition that lets every test claim every core and makes several thrash.
Two real defects in the parallelism work appeared only under a non-default
cap: an assertion whose two outcomes collided on the same number, and a
policy clause reading omp_get_num_procs() where it needed
omp_get_max_threads(), which planned sixteen threads under
OMP_NUM_THREADS=2. One environment cannot find that class.

So the C suite now also runs at one thread and at three. One is where a
reduction or a per-thread buffer can be skipped entirely; three is odd,
divides no chunk count evenly, and -- unlike another power of two -- cannot
collide with the plan fallback that made the test above pass by luck. At
one thread the suite reports 438 passed and 1 skipped: test_plan.c:607
needs two threads to tell "the worker inherited the plan" from "the worker
fell back", and skipping is the right answer there.

The new tests assert the property rather than the YAML spelling, and each
was mutation-tested: commenting out the tooling step, moving --threads 1 to
2, moving --threads 3 to 4, and dropping the OMP_NUM_THREADS pin each fail
exactly one test and leave the other six passing. They prove a command is
present outside a comment, not that the job is reached or that it passed --
the same limit the thread-plan wiring line turned out to have, stated in
the file rather than discovered later.
The gate grew a line in 7f3717e and the table here never gained the row,
so the one check that would have caught a region running at one thread per
core regardless of the plan was undocumented in the file that explains the
gate.

Two counts beside it had drifted with the suites they describe: the C half
is 439 tests, not 331, and `exact` holds 416 quantities, not 351. Neither
number is recorded anywhere the gate reads, which is why they could drift
in the first place.
The new thread-environment job failed on its first run, which is the job
working: under OMP_NUM_THREADS=3 two tests in test_runtime.py failed, one
asserting the plan never exceeds availability and one asserting a pinned
BEAM_AVAILABLE_CORES drives the plan.

Neither is a defect in BEAM. Clause 2 of the policy
(src/native/core/plan.c) treats OMP_NUM_THREADS as the caller deciding,
exactly as set_threads() does, so the runtime's own answer stands and
availability does not revise it -- a plan of 3 against an availability of 2
is the documented behaviour, and under a cap of 16 the plan is 16, wider
than max_width, for the same reason.

Both tests describe clause 3, the path that reads the free cores, and
neither said so. They passed for years because no run had ever set the
variable: a test whose precondition is "the environment nobody varied" is
not wrong, it is silently narrow, and the fix is to state the precondition
rather than to weaken the assertion.

So each skips when the plan reports omp_num_threads, which is the value the
policy actually read rather than a second guess at os.environ. This is how
the C side already does it: test_the_default_scope_stands_aside_for_omp_
num_threads skips unless the variable is set at process start, and pins
availability to 64 to prove clause 2 ignores it. Between them the two
clauses are covered in the environments where each applies -- clause 3 on
the six matrix legs where the variable is unset, clause 2 on the C side and
in the new job.

The pinned-availability test keeps its first assertion unconditional: that
BEAM_AVAILABLE_CORES is observable in the plan holds under every clause,
and only its effect on the width is clause 3's business.

Verified both ways: under OMP_NUM_THREADS=3 the two skip with their reasons
and the file's other 12 tests pass; with the variable unset both run and
pass, so the guard is conditional rather than a disable. The test ids do
not move -- collection is byte-identical to pytest_cases.txt at 1243.
@MuteJester
MuteJester merged commit 30031e5 into main Nov 15, 2026
15 checks passed
@MuteJester
MuteJester deleted the ci/tooling-tests-and-thread-envs branch November 15, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant