Map overlap GPUs through infer placement - #19
Conversation
ea4cdc2 to
541844b
Compare
541844b to
32a5485
Compare
JunzheJoe
left a comment
There was a problem hiding this comment.
Verified against HEAD and the coordinator's mapping convention:
- The
gpu_to_enginegrouping (placement order,per_engineGPUs per engine) matches the C6 first-build contiguous-mapping convention the coordinator uses (device_mapping[dp_rank*tp_size : ...]), so local indices agree across both consumers. - The filter semantics are the actual bug fix: old code computed arithmetic indices for train GPUs even when they were outside the infer pool (M11.2 P2 pool [2,3] → bogus local index →
states.get(i, '?')never converges → full timeout burned). Now non-overlap GPUs are excluded and fully-disjoint topologies return early — correct, since there is nothing to wait for. - Fallback
range(rollout_num_gpus)preserved; the divisibility raise is a fail-fast improvement over silently mis-grouping. - The
"actor_infer"literal is consistent with what HEAD still does at the othercluster_device_mappingssites (the merged #30 rename covered cluster IDs, not these mapping keys), so no regression — the literal cleanup can stay a separate batch.
Sequencing note: #19/#20/#21 all edit _wait_for_overlap_engines_offloaded and #19/#20 collide on the same warn-block lines. Suggest landing this one first (it's the refactor), then #20 rebases its raises onto it, then #21. LGTM.
|
Follow-up after learning (via #21's closure) that
My approval stands for the mapping fix itself; needs the rebase onto the rewritten function before it can land. |
|
I agree with Joe, MILES_MAX_RESIDUAL_GPU_MEM_GB causes merge conflicts, but the fix is good |
Context
_wait_for_overlap_engines_offloadedreceives physical GPU IDs from the scheduler, while the rollout manager expects local engine indices within the pipeline's inference placement. The previous conversion assumed the inference GPUs were contiguous and derived engine indices by subtracting the first GPU ID.Change
_resolve_overlap_infer_enginesunder the file's Helpers sectionactor_inferplacementoverlap_engine_indicesandoverlap_gpu_idsfor the wait logicrollout_num_gpus_per_engineValidation
python -m py_compile rlix/pipeline/miles_pipeline.pygit diff --check