Skip to content

synchronization: don't lose a release() when counting_semaphore::wait_until times out - #7802

Merged
hkaiser merged 1 commit into
TheHPXProject:masterfrom
aneek22112007-tech:fix-counting-semaphore-timeout-lost-signal
Oct 20, 2026
Merged

hkaiser merged 1 commit into
TheHPXProject:masterfrom
aneek22112007-tech:fix-counting-semaphore-timeout-lost-signal

Conversation

@aneek22112007-tech

Copy link
Copy Markdown
Contributor

Proposed Changes

  • Report a timeout from counting_semaphore::wait_until only when value_ < count.
  • When a credit is already available after cond_.wait_until returns thread_restart_state::timeout, consume it and return true.
  • Add test_wait_until_timeout_does_not_lose_signal to libs/core/synchronization/tests/unit/counting_semaphore_cpp20.cpp.

Any background context you want to provide?

A waiter whose deadline expires can still be queued on the semaphore's condition variable when release() runs. execution_agent::sleep_until has already seen was_reset_ == false and returned thread_restart_state::timeout, but the waiter has not yet reacquired the semaphore spinlock. It leaves the condition-variable queue only after that relock (reset_queue_entry runs once unlock_guard has taken the mutex again, and only if notify_one has not already cleared ctx_).

release() then does value_ += count and notify_one() pops this waiter, because it is still at the front of the queue. The waiter relocks, observes timeout, and used to return false without looking at value_ again. The released credit stayed unclaimed, and the next waiter was never notified. try_acquire_for / try_acquire_until, and binary_semaphore (a counting_semaphore<1>), could lose a release() and leave a later waiter blocked even though a credit was available.

timed_mutex::try_lock_until already breaks out of its wait on timeout and then rechecks owner_id_ before reporting failure. counting_semaphore::wait_until now does the same for the count: a timeout is a failed wait only when no credit is available. A thread_restart_state::signaled wakeup still falls through and re-evaluates the predicate, which is the behavior #7617 established.

The regression test holds the semaphore lock across the waiter's deadline and calls signal afterwards, so the notification happens only after the waiter has already timed out. It requires the waiter to take the credit (state == 3) and requires a following try_acquire to see no leftover count.

Without this change, counting_semaphore_cpp20_test failed 10/10 with --hpx:threads=4 and 10/10 with --hpx:threads=2 (state stayed 2, and try_acquire found a leftover credit). With this change, counting_semaphore_cpp20_test passed 20/20 at 2 threads, 20/20 at 4 threads, and once at 1 thread. binary_semaphore_cpp20_test passed 3/3 at 2 threads and 3/3 at 4 threads. counting_semaphore_test passed 3/3 at 2 threads and 3/3 at 4 threads.

Checklist

  • I have added a new feature and have added tests to go along with it.
  • I have fixed a bug and have added a regression test.
  • I have added a test using random numbers; I have made sure it uses a seed, and that random numbers generated are valid inputs for the tests.

…_until times out

A waiter whose deadline expires can still be queued on the semaphore's condition variable when release() runs: it has already decided it timed out but has not yet reacquired the lock. release() increments the count and notify_one() hands the wakeup to that waiter, which then returned false from wait_until without looking at the count again. The released credit stayed unclaimed while the next waiter was never notified, so try_acquire_for/try_acquire_until (and binary_semaphore) could lose a release() and leave a waiter blocked despite an available credit.

Only report a timeout when no credit is available, the same way timed_mutex::try_lock_until rechecks its predicate after a timeout.

Add a regression test that holds the semaphore lock across the waiter's deadline and signals afterwards, so the waiter is notified after it has already timed out.

Signed-off-by: aneek22112007-tech <[email protected]>
@StellarBot

Copy link
Copy Markdown
Collaborator

Can one of the admins verify this patch?

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@coderabbitai

coderabbitai Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 47ca7c8a-8379-4633-ba34-d36b643769bc
📥 Commits

Reviewing files that changed from the base of the PR and between 97c7e25 and 9e25d20.

📒 Files selected for processing: 2
  • libs/core/synchronization/src/detail/counting_semaphore.cpp
  • libs/core/synchronization/tests/unit/counting_semaphore_cpp20.cpp

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Summary

Summary by CodeRabbit

  • Bug Fixes
    • Improved timed semaphore waits so resources that become available around the timeout boundary are correctly recognized and consumed.
    • Prevented valid signals from being left behind when a waiter is waking near its deadline.

wait_until now checks resource availability after a timed wait. A new unit test verifies that a signal arriving after the deadline is consumed by the waiting thread.

Changes: Semaphore timeout handling

Layer / File(s) Summary
Timeout wakeup and signal consumption
libs/core/synchronization/src/detail/counting_semaphore.cpp, libs/core/synchronization/tests/unit/counting_semaphore_cpp20.cpp
wait_until returns false after a timeout only when fewer than the requested resources are available. The new test checks that a signal arriving after the deadline is acquired and leaves no permit behind.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to 9e25d

The change preserves a permit when a timed wait reports timeout at the same time a signal becomes available. No actionable merge-blocking risk remains after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main bug fix: preventing counting_semaphore::wait_until from losing a release() after timeout.
Description check Passed The description directly explains the wait_until behavior change, the race condition, the regression test, and validation results.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@hkaiser hkaiser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@hkaiser hkaiser added this to the 2.0.0 milestone Oct 20, 2026
@hkaiser
hkaiser merged commit 71e510b into TheHPXProject:master Oct 20, 2026
84 checks passed
@hkaiser hkaiser mentioned this pull request Oct 20, 2026
1 of 2 tasks
@hkaiser

hkaiser commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

@aneek22112007-tech, the new test_wait_until_timeout_does_not_lose_signal in counting_semaphore_cpp20 is failing on Jenkins since this was merged. It fails in the gcc-12-cuda-12 builds of several unrelated pull requests, e.g. https://cdash.rostam.cct.lsu.edu/test/94148874 (#7784), https://cdash.rostam.cct.lsu.edu/test/94174628 (#7787), https://cdash.rostam.cct.lsu.edu/test/94193170 (#7809) and https://cdash.rostam.cct.lsu.edu/test/94228047 (#6514), always with state.load() == 3 failing with '2' != '3', followed by !left_over.

I think the test assumes something about timing that the CI doesn't guarantee: the main thread has to lock mtx within the 10 ms between the waiter's state = 1 and its deadline. If the main thread is scheduled late (these builds run many tests in parallel, with --hpx:bind=none), the waiter's deadline expires while nobody holds mtx. It relocks it right away, finds no credit, and correctly returns false (state 2). The main thread then locks mtx, spins and signals with no waiter, which leaves the credit over. So the semaphore behaves correctly; it's the test that didn't produce the situation it checks for.

Could you please make the test independent of this? For instance, the main thread could check, right after it has locked mtx, whether the deadline has already passed. If it has, the iteration didn't exercise the race: just signal, join, and repeat it instead of checking the results. A longer deadline alone would only make the failure rarer.

hkaiser added a commit that referenced this pull request Oct 21, 2026
Describe the PRs merged since the release branch was rebuilt on master cba138f (#7675, #7720, #7723, #7725, #7726, #7735, #7736, #7738, #7740, #7741, #7749, #7754, #7755, #7757, #7759), since it was rebuilt on master 97c7e25 (#7750, #7752, #7762, #7771, #7781, #7782, #7791, #7793), and since it was rebuilt on master 67e52ba (#7718, #7739, #7751, #7753, #7756, #7770, #7797, #7800, #7802, #7804, #7805). Regenerate the closed issue and pull request lists for release-2.0.X at a55fd37 against the v1.11.0 tag (765 PRs, 157 issues).

Signed-off-by: Hartmut Kaiser <[email protected]>
hkaiser pushed a commit that referenced this pull request Oct 22, 2026
…pendent of scheduling

The regression test added in #7802 assumed the main thread acquires the
mutex before the waiter's 10 ms deadline. On loaded CI nodes (Jenkins
gcc-12-cuda-12) the main thread can be scheduled after the deadline; the
waiter then correctly times out without the credit, and the test failed
with state 2 and a left-over credit even though the semaphore behaved
correctly. Record whether the lock was taken before the deadline, always
signal past the deadline, join and drain the semaphore, and only check
the iterations that actually exercised the timeout/signal race, with an
upper bound on attempts so the test cannot run forever.

Signed-off-by: aneek22112007-tech <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants