Repository navigation
Conversation
get_default_dist_backend() returned the historical "nccl" for every device that is not Ascend NPU, but get_accelerator_device_type() does select XPU, so an XPU run reached init_process_group(backend="nccl") and could not start on a build without NCCL. Resolve XPU to XCCL (legacy oneCCL name "ccl" where that is the only probe the build exposes). Fixes modelscope#3777. Signed-off-by: li-lizhe <147392333@qq.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_default_dist_backend()returns"nccl"for every device that is not Ascend NPU. Sinceget_accelerator_device_type()does select XPU, an XPU run reachesinit_process_group(backend="nccl")and cannot start at all on a build without NCCL(
RuntimeError: Distributed package doesn't have NCCL built in) - the same class of failure #3771fixed for Ascend NPU. This is the follow-up tracked in #3777, which was kept out of #3771 only
because that change had no XPU hardware to exercise the new path.
Change
_ACCELERATOR_DIST_BACKENDS:"xpu": ("xccl", "ccl");torch.distributed.is_xccl_available()wins, a build thatonly exposes the former oneCCL name (
is_ccl_available()) resolves to"ccl", and a build thatexposes no probe keeps the first (canonical) name,
"xccl"- the same convention already used forHCCL on Ascend;
on is gone;
tests/test_distributed_device_setup.py: the case that pinned XPU to the historical"nccl"isreplaced by three: XPU resolves to
"xccl","xccl"wins when both probes report available, and"ccl"is used when only that probe does.Fixes #3777.
Verification
get_default_dist_backend()for XPU"nccl"is_xccl_available()true"xccl"is_ccl_available()true"ccl""xccl"(first/canonical name)Local CPU-only venv (torch 2.8.0+cpu, pytest 9.1.1),
pytest tests/test_distributed_device_setup.py:10 passed with this change; with only the
_ACCELERATOR_DIST_BACKENDSline reverted and the newtests kept: 3 failed, 7 passed (the three new XPU cases), so the cases do lock the behaviour.
Not tested: a real 2-rank
torchrunon Intel XPU with XCCL. No XPU device exists in thisenvironment, so the runtime path is only verified as far as the backend selection itself - I am
flagging that here rather than claiming an XPU run. The CUDA and Ascend NPU paths are untouched. If
you would rather hold this until someone with XPU hardware confirms it end to end, the change is a
one-line mapping plus tests and is easy to revert.