[xpupti] Support more than 1 device in ptiMetricsScopeConfigure - #1479
[xpupti] Support more than 1 device in ptiMetricsScopeConfigure#1479aostrowski-hbn wants to merge 6 commits into
Conversation
|
@pytorchbot label ciflow/xpu |
|
The ciflow label(s) ciflow/xpu will be added, but CI won't be triggered until the workflows are approved (scroll to the bottom of this page). Please ping one of the reviewers if you do not have access to approve and run workflows. |
|
|
8f18fc8 to
f2275e5
Compare
|
|
||
| #if PTI_VERSION_AT_LEAST(0, 18) | ||
| if (requestedDevices.empty()) { | ||
| // Default: profile every available device (PTI auto-detect mode). |
There was a problem hiding this comment.
This comment was inaccurate. When requested devices is 0, it will profile whichever devices the workload actually uses.
Note: you may also want to update the PR body.
|
One comment to the Testing description: "Auto-detect initializes 16 per-device metrics handlers" is true because the 16-tile PVC test workload dispatched to all 16 tiles — not because auto-detect unconditionally profiles all devices. |
|
Nit: The PTI implementation validates uniformity for any multi-device configure: So on a mixed-GPU host, both explicit multi-device selection and auto-detect (if the workload touches >1 non-uniform device) will fail inside |
| TEST_F(XpuptiScopeProfilerConfigTest, DevicesParsedList) { | ||
| KN::Config cfg; | ||
| EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_METRICS = metric1")); | ||
| EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_DEVICES = 0, 2, 3")); |
There was a problem hiding this comment.
nit: would be nice to add a test for testing empty "tokens", for example:
EXPECT_TRUE(cfg.parse("XPUPTI_PROFILER_DEVICES = 0, ,2,"));
Trailing comma + doubled comma: empty tokens must be skipped, not parsed as index 0 or rejected as invalid integers.
Additionally, what if there are duplicated indices: XPUPTI_PROFILER_DEVICES=0,0,2?
Empty list means all devices (PTI auto-detect); a comma-separated list selects an explicit device-index subset, parsed with a C++20 views::filter|transform pipeline. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Remove the hardcoded device_count=1 in enableScopeProfiler. On PTI >= 0.18 (PTI-363) default to auto-detect (all devices) and honor an explicit XPUPTI_PROFILER_DEVICES subset; on older PTI keep single-device behavior and reject a >1-device request with a clear error. Add selectDeviceHandles(std::span<const pti_device_handle_t>, std::span<const int>): a C++20 ranges gather (find_if bounds check + transform) mapping requested indices to device handles. Host-only gtests cover the config key and the helper. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add PerKernelScopeExplicitDevice0 driving the selectDeviceHandles -> ptiMetricsScopeConfigure(handles,count) branch on real hardware; parameterize RunTest with an optional XPUPTI_PROFILER_DEVICES value. PYTORCHDGQ-8150. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- reword auto-detect comment and config print ("auto", not "all")
- skip empty tokens and drop duplicate XPUPTI_PROFILER_DEVICES indices, with tests
- document device-model uniformity requirement
- raise via KINETO_THROW (RAWTHROW lint)
f2275e5 to
c0cd7ec
Compare
|
@Rogersyp solved your comments, can you take another look? |
Rogersyp
left a comment
Rogersyp
left a comment
There was a problem hiding this comment.
LGTM! Thank you for addressing my comments.
|
@EikanWang, @jgong5, @gujinghui please review |
PerKernelScopeExplicitDevice0 exercises the same RunTest("true", 314)
path as PerKernelScope, which is already skipped: PTI returns
PTI_SUCCESS from every ptiMetricsScope* call while silently dropping a
device whose StartProfiling failed, so the expected scope metric
activities never arrive and the test fails on a value mismatch that
hides the real cause. Skip it for the same reason, pointing at the same
tracking issue.
Also apply clang-format to the files this branch touches, so the CLANGFORMAT
linter passes.
3 participants
Enables the XPU Scope Profiler to collect hardware metrics on more than one XPU. Previously
enableScopeProfilerhardcodeddevice_count=1in theptiMetricsScopeConfigurecall, so metrics were only ever collected on a single card even on multi-GPU hosts.What changed
devices=nullptr, count=0): profiles all available devices.XPUPTI_PROFILER_DEVICES=0,2,3selects an explicit device subset via a newselectDeviceHandleshelper (C++20std::span+std::rangesgather with bounds validation).PTI_VERSION_AT_LEAST(0, 18): multi-device support inptiMetricsScopeConfigurelanded in PTI 0.18. On older PTI, single-device behavior is preserved and a request for >1 device is rejected with a clear error.Device attribution needs no change: each scope record is already correlated to its kernel activity (and thus device/resource) via
_kernel_id.Testing
Verified on a 16-tile Intel Data Center GPU Max (PVC) host built against PTI 0.18:
XpuptiScopeProfilerConfigTest): 8/8 pass — config-key parsing +selectDeviceHandlesbounds validation.XpuptiScopeProfilerTest): 3/3 pass, including a newPerKernelScopeExplicitDevice0exercising the explicit-subset path.