fix(rlix-scheduler): warn on GENERATION request without demand signal - #34
Open
JunzheJoe wants to merge 1 commit into
Open
fix(rlix-scheduler): warn on GENERATION request without demand signal#34JunzheJoe wants to merge 1 commit into
JunzheJoe wants to merge 1 commit into
Conversation
The gap-ratio planner skips a pipeline whose rollout_open_pipelines entry has no positive step_target_estimate AND no stored progress (the bootstrap-estimate branch in planner.plan_generation_gap_ratio). Such a request is never signaled and its caller blocks forever in request_gpus — silently. The hazard was documented only in MilesPipeline's docstring. Warn at enqueue time (request_gpus + notify_release_then_request_gpus) via a shared helper. Warn-only, no behavior change: some backends legitimately request first and report progress later, so rejecting outright could break them.
zhenyulincs
approved these changes
Aug 2, 2026
zhenyulincs
left a comment
Collaborator
There was a problem hiding this comment.
Verified the finding end-to-end before approving:
- The planner skip condition is real (
planner.py:199-209): with no stored progress (step_target <= 0) and no positive estimate, the pipeline iscontinue-skipped — its pending request is never signaled, andrequest_gpusblocks without timeout by design. - Semantic alignment is exact: the warn helper keys off the second element of
_pipeline_progress_totals_locked(total_required), which is the same quantity the planner's skip branch tests. No drift risk between the warning and the actual skip behavior. - Coverage is complete: the only two write sites of
rollout_open_pipelines[...]in the repo are the two enqueue points this PR instruments, and both helper calls sit insideself._lock. - Why nothing hits this today: every in-repo caller already defends itself —
MilesPipeline._init_phase_b_infercomputesmax(rollout_batch_size * n_samples_per_prompt, 1)(its comment cites exactly this hazard), the steady-state re-request threads the demand signal'sstep_target, and the ROLL backend computes its own estimate. So the trap is currently unreachable, guarded only by tribal knowledge copied into each caller. This PR moves that guard to the scheduler side where it belongs. - Ran the new test file (4/4) and the full suite on the PR head — green.
Warn-only is the right call given the request-then-report flows some backends legitimately use, and the shared helper gives a single flip point if we later decide to hard-reject.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Review finding S12-EST-NONE-HANG (F1-F12 code review, scheduler deep-dive).
request_gpus(GENERATION, step_target_estimate=None)is accepted and writesNoneinto the durablerollout_open_pipelinesregistry. On the planner side (planner.plan_generation_gap_ratio, bootstrap-estimate branch), a pipeline with no positive estimate and no stored progress is skipped entirely — its pending request is never signaled, and the caller blocks forever insiderequest_gpus(which waits without timeout by design). No log line points at the cause; today the hazard is documented only inMilesPipeline._request_cluster_gpus's docstring.This adds a
logger.warningat both enqueue sites (request_gpusandnotify_release_then_request_gpus) via a shared_warn_gen_request_without_demand_lockedhelper, naming the exact skip condition and the likely outcome.Warn-only, no behavior change: some backends may legitimately enqueue first and publish progress later, so rejecting outright could break request-then-report flows. If we later decide to hard-reject, this helper is the single place to do it.
Tests
tests/test_scheduler_gen_request_warning.py— warn / no-warn matrix:Full suite: 69 passed, 3 skipped.