Repository navigation
Fetch next request doesn't return just-added or reclaimed requests in shared RQ #808
Description
Context
Discovered during review of PR #803 (discussion).
The test_request_reclaim_functionality[shared] integration test is flaky because fetch_next_request() does not immediately return requests that were just added or reclaimed when using the shared request queue access mode. In the single mode, requests are added directly to the local queue_head deque and are available right away, but the shared client cannot do that.
Problem
When using shared request queue mode, there is a propagation delay between:
- Adding a request (or reclaiming one) via the API
- That request becoming visible in
fetch_next_request()
This means users of the shared request queue mode may call fetch_next_request() and get None even though a request was just added or reclaimed.
Expected behavior
fetch_next_request() should reliably return requests that have been added or reclaimed, even in shared mode, without requiring the caller to implement their own retry/polling logic.
Current workaround
Callers need to poll fetch_next_request() with retries and delays to handle the propagation delay.
Activity
janbuchar commentedon Feb 20, 2026
Just noting, isn't the expected flow more like
while not rq.is_finished(): next_request = rq.fetch()? In that case, the fix (which will probably be just "wait for some reasonable time ofrq.is_empty() and rq.locked_request_count == 0") will probably fit better into theis_finishedmethod.test: Add exponential backoff for shared RQ propagation delay in inte…
test: raise `call_with_exp_backoff` retry budget for flaky shared RQ …
test: fix flaky `test_get_request_by_unique_key[shared]` (#873)
test: address review findings on shared-mode ordering relaxation
test: fix flaky request ordering tests in shared RQ access mode (#931)
test: deflake request queue fetch-and-handle integration tests (#991)
[-]Shared request queue: fetch_next_request does not immediately see newly added/reclaimed requests[/-][+]Fetch next request doesn't return just-added or reclaimed requests in shared RQ[/+]test: poll the first fetch in the shared RQ in-progress test (#1105)
vdusek commentedon Aug 31, 2026
Closing as this was probably resolved by #1087