We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
From #138, multi-GPU doesn't work out of the box with SLURM
--ntasks-per-node = 4
#SBATCH --gres=gpu:4 #SBATCH --ntasks-per-node=4
srun python ...
In addition to LOCAL_RANK, check SLURM_LOCALID in losses.py and dali.py. See Lightning-AI/pytorch-lightning#6797.
losses.py
dali.py
The text was updated successfully, but these errors were encountered:
ksikka
No branches or pull requests
From #138, multi-GPU doesn't work out of the box with SLURM
1. Add docs that you need to do the following
--ntasks-per-node = 4
in your submit script, replacing 4 with the number of GPUs you want to use.srun python ...
instead of just python.2. Fix code: LOCAL_RANK not set in SLURM
In addition to LOCAL_RANK, check SLURM_LOCALID in
losses.py
anddali.py
. See Lightning-AI/pytorch-lightning#6797.The text was updated successfully, but these errors were encountered: