Repository navigation
Sense contention on macOS and Windows, not just on Linux - #12
Merged
Merged
Conversation
beam_available_cores() read /proc/loadavg inside #if defined(__linux__) and
nothing else, so on macOS and Windows -- two of the three platforms BEAM
ships wheels for -- the runnable count was always unknown. Unknown means
"assume no contention", so clause 3 planned eight threads whatever the
machine was doing. Phase 0's catastrophe, 301x slower than serial on a
saturated node, was defended against on one platform of three.
The obvious portable answer is already ruled out by the measurements in
avail.h: the one-minute load average is wrong by -6 at eight competitors,
and half a second after twelve spinners exited it still read 9.22 while the
runnable count had returned to 2. A plan built on it "would serialise every
call for a minute after BEAM's own previous call finished". Both platforms
have getloadavg(). It is not the answer.
So they difference two cumulative CPU-time readings instead:
competitors = ceil((system_busy_ns - own_ns) / wall_ns), clamped
macOS reads host_statistics(HOST_CPU_LOAD_INFO), getrusage(RUSAGE_SELF) and
clock_gettime(CLOCK_MONOTONIC); Windows reads GetSystemTimes,
GetProcessTimes and the performance counter. One syscall each and no sleep,
which the cost budget demands: this runs at every entry point with no
caching, and Linux answers in microseconds.
own_ns is subtracted, and that is the whole design. Without it a window
spanning BEAM's own eight-thread region reads as eight cores busy, so every
call would narrow the next BECAUSE OF THE PREVIOUS ONE -- the same feedback
loop the load average was rejected for. Linux escapes it for free, the
runnable count being instantaneous: by the time it reads, the workers have
exited. A sampled estimate has to subtract its own usage to measure the same
quantity.
The three state rules live in beam_competitors_from_samples rather than in
platform code, so they are testable on any machine. A thread's first reading
is only a baseline. A window below twenty milliseconds reports unknown and
leaves the baseline to age -- replacing it per call is the bug that would
make the mechanism useless, since a tight loop of short calls would restart
the window and never reach twenty milliseconds, on exactly the workload that
needs it. A reading that moves backwards resets rather than subtracting into
an unsigned underflow.
Every failure path returns -1, which is what these platforms returned on
every call before, so this cannot be worse than what it replaced; it can
only start being right sooner. It is second best and says so: a window
rather than an instant, so it lags, and the lag lands on the safe side.
Planning narrow costs about 2x and planning wide cost up to 3814x, which is
also why a fraction of a core rounds up.
Eleven tests, four of them mutated against the real suite: advancing the
baseline on a short window, dropping the backwards guard, dropping the
subtraction of our own CPU time, and rounding down instead of up each fail
exactly the test that names the property.
Two defects in the first draft, found by re-reading rather than by a test,
and neither reachable from Linux. mach_host_self() hands out a send right
per call and was leaking one Mach port reference per entry point. And
counter * 1000000000 on the performance counter overflows int64 after about
fifteen minutes of uptime at 10 MHz -- longer than a CI runner lives, so it
would have passed green here and returned nonsense on a machine up since
Tuesday.
Two stale claims corrected on the way. avail.h credited "the policy's own
halving" with the conservative fallback; plan.c stopped halving when the
ceiling form replaced it, and the ceiling does that work now. And
test_avail.c said "no /proc, as on Windows" while describing the unknown
path as not a special case anywhere -- it was the gap, in a comment.
Verified: C 450 passed, Python 1241 passed and 2 skipped, gate exit 0 with
no FAIL and no note lines, every recorded number byte-identical, extension
rebuilt from a2cc902b to 334d37e9. The macOS and Windows readers do not
compile on this machine; CI is their first build and their first run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beam_available_cores()read/proc/loadavginside#if defined(__linux__)andnothing else. On macOS and Windows — two of the three platforms BEAM ships wheels
for — the runnable count was therefore always unknown, which means "assume no
contention", which means clause 3 planned eight threads whatever the machine was
doing. Phase 0's catastrophe, 301x slower than serial on a saturated node, was
defended against on one platform of three.
The obvious answer was already ruled out
Both platforms have
getloadavg().avail.hcarries the measurement thatdisqualifies it:
loadavg[0]nr_runningSo these platforms need a different signal, not a portable spelling of the same
one.
What they do instead
macOS reads
host_statistics(HOST_CPU_LOAD_INFO),getrusage(RUSAGE_SELF)andclock_gettime(CLOCK_MONOTONIC). Windows readsGetSystemTimes,GetProcessTimesand the performance counter. One syscall each, no sleep — thecost budget allows nothing more, since this runs at every entry point with no
caching and Linux answers in microseconds.
own_nsis subtracted, and that is the whole design. Without it, a windowspanning BEAM's own eight-thread region reads as eight cores busy, so every call
would narrow the next because of the previous one — the same feedback loop the
load average was rejected for. Linux escapes it for free:
nr_runningisinstantaneous, so by the time it reads, BEAM's workers have exited. A sampled
estimate has to subtract its own usage to measure the same quantity.
Three state rules, kept testable
They live in
beam_competitors_from_samples, not in platform code, so they can bepinned on any machine:
Replacing it per call is the bug that would make the whole mechanism useless —
a tight loop of short calls would restart the window and never reach 20 ms, on
exactly the workload that needs it.
underflow.
Every failure path returns -1, which is what these platforms returned on every
call before. This cannot be worse than what it replaced; it can only start being
right sooner.
Honest about being second best
It measures a window rather than an instant, so it lags where
nr_runningdoesnot. The lag lands on the safe side — planning narrow costs about 2x, planning
wide cost up to 3814x — which is also why a fraction of a core rounds up. Two
things it cannot do, both documented: system CPU time covers every CPU while
allowedmay be a subset (the runnable count makes the same approximation), and abusy fraction saturates at 100% where a count does not, so it cannot tell 48 busy
cores from 48 cores with 200 runnable threads. Both land on
available = 1, whichis the answer that matters.
Two defects CI would not have caught
Found by re-reading the code, not by a test — neither line is reachable from Linux:
mach_host_self()hands out a send right on every call, andthis runs at every entry point: one leaked port reference per BEAM call.
Invisible to any test, fatal to a long-lived process. Now paired with
mach_port_deallocate.counter * 1000000000overflows int64 afterroughly fifteen minutes of uptime at the usual 10 MHz. A CI runner lives
about that long, so the obvious one-liner passes green here and returns nonsense
on a developer's machine. Now split into seconds plus remainder.
Two stale claims corrected
avail.hcredited "the policy's own halving" with supplying the conservativefallback.
plan.cstopped halving when the ceiling form replaced it; the ceilingdoes that work now. And
test_avail.csaid "no /proc, as on Windows" whiledescribing the unknown path as "not a special case anywhere" — that was the gap,
sitting in a comment.
Verified
byte-identical — 78 numeric keys, 58 bit-identical math values, the saved-model
sha256, thread independence, 416 exact quantities
a2cc902b…→334d37e9…the backwards guard, dropping the own-time subtraction, and rounding down each
fail exactly the test that names the property
availablestill tracks instantaneously (15 then 14of 16 cores between two reads)
Baselines moved by inventory only —
c_cases.txt439 → 450 andc_symbols.txt261 → 263 — with a
RERECORDED.mdsection naming all thirteen lines.The macOS and Windows readers do not compile on the machine that wrote them.
CI is their first build and their first run, which is the point of having three
platforms in it.