Skip to content

Silences NCCL warning #1055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Oct 5, 2025
Merged

Silences NCCL warning #1055

Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
352eb0f
Loads model on device
finbarrtimbers Oct 3, 2025
f431c0d
Update open_instruct/grpo_fast.py
finbarrtimbers Oct 3, 2025
5948e8a
Set conditional device map
finbarrtimbers Oct 3, 2025
fcda1d6
Set CUDA device before init_process_group to silence NCCL warning
finbarrtimbers Oct 3, 2025
c5a2c33
Set CUDA device immediately before DeepSpeed init to silence NCCL war…
finbarrtimbers Oct 3, 2025
160b43c
trying to fix it
finbarrtimbers Oct 3, 2025
a09f344
Merge branch 'main' into silence-warning2
finbarrtimbers Oct 3, 2025
ca46601
Removed unneeded code.
finbarrtimbers Oct 3, 2025
eadadc7
Cleaned up PR.
finbarrtimbers Oct 3, 2025
8c1d55d
Cleaned PR
finbarrtimbers Oct 3, 2025
dbc5a40
Cleaned PR
finbarrtimbers Oct 3, 2025
d686ea6
Undid changes to open_instruct/utils.py
finbarrtimbers Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions open_instruct/grpo_fast.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -625,6 +625,15 @@ def load(self, path: str, map_location=None):
np.random.seed(worker_seed)
random.seed(worker_seed)

torch.distributed.init_process_group(
backend="nccl",
init_method="env://",
world_size=self.world_size,
rank=self.rank,
timeout=timedelta(minutes=args.backend_timeout),
device_id=torch.device("cuda", self.local_rank),
)

deepspeed.init_distributed(timeout=timedelta(minutes=args.backend_timeout))

ds_config = get_train_ds_config(offload=False, adam_offload=False, stage=args.deepspeed_stage, bf16=True)
Expand Down