Repository navigation
Point a race detector at the OpenMP code, and stop nested regions bypassing the plan - #14
Merged
Merged
Conversation
BEAM has 93 parallel regions and no race detector had ever been pointed at one of them. The two checks that exist are static: thread_count_clauses proves every region NAMES a thread count, the GIL scan proves every GIL-releasing region RESOLVES a plan. Neither says anything about what two threads do to the same address. `make tsan` builds clang + libomp into build-tsan/, `tools/qa/run c --tsan` runs it. 33 suites, 366 tests, 0 races, 782 s. Three things had to be fixed before that number existed, and each one would have produced a confident wrong answer on its own. Archer is not optional. libomp carries no sanitiser annotations, so without an OMPT tool TSan sees no happens-before edge between any two OpenMP threads: every implicit barrier is invisible and every region whose result somebody later reads is a race. Same binary, one suite at 8 threads: 783 reports with Archer off, 0 with it on. Under GCC and libgomp, where there is no Archer at all, 1186. So `--tsan` refuses to run when it cannot find libarcher.so rather than reporting a thousand races, and the last test in verification/tests/test_tsan.py runs a suite with the tool unset and requires the reports to come back -- a clean run is only evidence if a dirty one is reachable. target_clones and TSan cannot coexist. The attribute emits an IFUNC resolver, the dynamic loader calls it during startup relocation before the sanitiser runtime initialises, and clang instruments the resolver, so __tsan_func_entry runs against uninitialised state and the process dies inside ld.so before main, printing nothing. Nine of 35 suites: exit 139, empty log. The three guards now carry `&& !BEAM_TSAN`, which single-versions those functions under instrumentation and leaves the shipped build untouched -- BEAM_TSAN preprocesses to 0 without the flag, and the forced rebuild gives a byte-identical libbeam.a and .so. TSan refuses to start under an address-space ceiling: it maps shadow memory across the address space and setrlimit() fails with 22. So verification/bounded cannot wrap it, and `--tsan` swaps it for a bare timeout plus max_allocation_size_mb and allocator_may_return_null=1, which do the same job from inside the process. Four suites stay uninstrumented, all four about the allocator, which is the one thing TSan replaces wholesale -- they fail on their own premise, reporting that the policy did not reach the binary and that resident memory is many times the model's size. The test derives that set from the sources (statm|VmRSS|mallopt|M_MMAP_THRESHOLD) instead of listing it, so a fifth such suite cannot be quietly dropped from the run. -mavx is on the sanitiser build because clang passes a 32-byte vector by value in vtrig.c, and with the clones off that is a -Wpsabi warning rather than an error. The flag makes the instrumented build agree with the shipped one about that ABI and keeps the log clean. Nothing in CI runs this; it needs a clang toolchain and today's answer is "no races on this machine today", not a standing one. verification/README.md says so, and says what the binaries mean if you run them any other way. Verified: 13 tooling tests, 11 of them written red first; C 452 tests passing, gate exit 0, every recorded number byte-identical, no inventory line moved by this commit.
beam_threads() consulted the thread-local plan and, failing that, the configured cap. Neither is what a nested region should get, and the failure mode is not that the plan gets multiplied -- it is that the plan is not consulted at all. The inner pragmas are encountered by the OUTER team's workers, and a worker holds no plan. The plan is thread-local deliberately: a process-global one would be visible to workers, but two BEAM calls on two Python threads would then overwrite each other's plan mid-call, which is the buffer-versus-team mismatch the mechanism exists to prevent. So a worker fell through to threads_cap(), which with no configured cap is omp_get_max_threads(). An 8-thread plan opened eight teams of SIXTEEN. Measured on one entropy() call with the plan pinned at 8 and nesting enabled from the environment with OMP_MAX_ACTIVE_LEVELS=2: 117 threads in the process against 9 with nesting off, 0.24 s against under a millisecond. After the fix, 9 either way. Phase 0 measured that class of oversubscription at 301x to 2275x worse than serial on a loaded cluster node, which is what makes it worth a behaviour change rather than a line in the documentation. omp_set_max_active_levels is not the answer. It is process-global state that two BEAM calls on two threads would fight over -- the same reason the plan is thread-local. A thread's own level is the one thing that is always true locally. ACTIVE level, not level. A team of one is an inactive region; it multiplies nothing, and reading omp_get_level() there would serialise the work inside a single-length sweep for no gain. test_an_inactive_region_does_not_lose_the_plan pins that choice against a later simplification. Safe for the scratch sizing, and in the direction that matters. A resolved count both sizes buffers and sets num_threads, so the dangerous disagreement is a buffer SMALLER than its team. This can only shrink the answer as the nesting deepens, and a region entered at a given level is sized by a read at that level or deeper, never by a shallower one. math/spectrum.c's per-thread scratch, allocated inside a nested call, was being sized for omp_get_max_threads() and used by one thread; eight suites on that path under AddressSanitizer, 116 tests, report nothing. One existing assertion changed, in test_a_plan_does_not_reach_another_thread, and RERECORDED.md says why at length: the test keeps its name and its property, the cap is still configured and still required to differ from the plan, but the fallback it recorded was the defect itself. Verified: C 454 tests passing, Python 1243 passed and 2 skipped, tooling 130 passed, gate exit 0 with every recorded number byte-identical -- including `thread counts`, the check this might have been expected to disturb, since the clauses are untouched and only what one of them evaluates to on a thread already inside a team has changed. The Python test is mutated against the real build: with the line commented out and the extension rebuilt it fails at 3.74 s with the process 109 threads above its plan; restored, 0.35 s. Two inventory files moved by three lines in total, named in RERECORDED.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two pieces of work on the parallelism core. They touch disjoint files, so each
commit carries its own evidence: the detector commit moves no recorded number
and no inventory line, and the nesting commit carries all three inventory
lines and the whole RERECORDED.md section.
Run a race detector over the OpenMP code (7fb368f)
93 parallel regions and no race detector had ever been pointed at one. The two
checks that exist are static --
thread_count_clausesproves every regionNAMES a count, the GIL scan proves every GIL-releasing region RESOLVES a plan.
Neither says anything about what two threads do to the same address.
make tsanbuilds clang + libomp intobuild-tsan/;tools/qa/run c --tsanruns it. 33 suites, 368 tests, 0 races. Three blockers had to fall first,
and each one would have produced a confident wrong answer on its own:
libompcarries no sanitiser annotations, sowithout an OMPT tool TSan sees no happens-before edge between any two OpenMP
threads: every implicit barrier is invisible and every region whose result
somebody later reads is a race. Same binary, one suite at 8 threads: 783
reports with Archer off, 0 with it on. Under GCC and libgomp, where there
is no Archer at all, 1186. So the mode refuses to run when it cannot find
libarcher.so, and the last test runs a suite with the tool unset andrequires the reports to come back -- a clean run is only evidence if a dirty
one is reachable.
target_clonesand TSan cannot coexist. The attribute emits an IFUNCresolver, the loader calls it during startup relocation before the sanitiser
runtime initialises, and clang instruments the resolver, so
__tsan_func_entryruns against uninitialised state and the process diesinside
ld.sobeforemain, printing nothing. Nine of 35 suites: exit 139,empty log. The three guards now carry
&& !BEAM_TSAN.memory across the address space and
setrlimit()fails with 22. Soverification/boundedcannot wrap it; the mode swaps it for a bare timeoutplus
max_allocation_size_mbandallocator_may_return_null=1.Four suites stay uninstrumented, all four about the allocator, which is the one
thing TSan replaces wholesale. The test derives that set from the sources
rather than listing it, so a fifth cannot be quietly dropped from the run.
Nothing in CI runs this: it needs a clang toolchain, and today's answer is "no
races on this machine today", not a standing one.
verification/README.mdsaysso, and says what the binaries mean if you run them any other way.
Ask for one thread when the region is a nested one (be77719)
beam_threads()consulted the thread-local plan and, failing that, theconfigured cap. The failure mode is not that the plan gets multiplied -- it is
that the plan is not consulted at all. The inner pragmas are encountered by
the OUTER team's workers, and a worker holds no plan (thread-local
deliberately: a process-global one would be visible to workers, but two BEAM
calls on two Python threads would then overwrite each other's plan mid-call).
So a worker fell through to
threads_cap(), which with no configured cap isomp_get_max_threads(). An 8-thread plan opened eight teams of sixteen.Measured on one
entropy()call, plan pinned at 8,OMP_MAX_ACTIVE_LEVELS=2:117 threads in the process against 9 with nesting off, 0.24 s against under a
millisecond. After the fix, 9 either way, entropy 17.6075259553 in all four
runs. Phase 0 measured that class of oversubscription at 301x-2275x worse than
serial on a loaded cluster node, which is what makes it worth a behaviour
change rather than a note in the docs.
omp_set_max_active_levelsis not the answer: process-global state that twoBEAM calls on two threads would fight over, the same reason the plan is
thread-local. A thread's own level is the one thing always true locally.
multiplies nothing, and
omp_get_level()would serialise a single-lengthsweep for no gain. A test pins that choice against a later simplification.
both sizes buffers and sets
num_threads, so the dangerous disagreement is abuffer SMALLER than its team. This can only shrink the answer as nesting
deepens.
One existing assertion changed, in
test_a_plan_does_not_reach_another_thread-- the part to scrutinise. The testkeeps its name and its property, and the cap is still configured and still
required to differ from the plan, but the fallback it recorded was the defect
itself. RERECORDED.md quotes the diff and says why.
Verification
thread counts (93 parallel regions, every one names a permitted thread count)is OK in the same gate run: the clauses are untouched, only what one of them
evaluates to on a thread already inside a team has changed.
The shipped build is unaffected by the detector commit --
BEAM_TSANpreprocesses to 0 without the flag, and a forced rebuild gives a
byte-identical
libbeam.aand.so. The 11 toolchain-free tooling tests passwith
build-tsan/absent and the other 2 skip, which is the state every runneris in.
Not addressed here
static.
beam_budget_concurrency'sper_item == 0early return skips thebeam_threads()cap below it. Latent: all 17 call sites pass a computed sizethat is 0 only for a 0-state model.