GPU usage is low.

[cinjon]

I have a model training on a 4 GPU machine that is only using 1 Gpu.

The output when starting it was:

You are using a CUDA device ('NVIDIA L4') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]

This makes me believe that it's seeing all of the devices, however not all of them are being used. I have devices: -1 in the config as well.

Running nvidia-smi shows that all of the GPUs were initiated because they all have at least 2mb in memory, but only the first one is doing any processing.

Any idea what's up? This seems straightforward...

[cinjon]

Ahh! If I set devices=4 explicitly, then this works. Setting it to -1 does not work.
Is there a way to set which devices to use?

[titu1994]

We use -1 throughout NeMo and if works for selecting all gpus in the current node - at least locally and slurm cluster.

https://lightning.ai/docs/pytorch/stable/accelerators/gpu_basic.html

It has this line -

# equivalent to `list(range(torch.cuda.device_count())) and `"auto"`
Trainer(accelerator="gpu", devices=-1)```

[cinjon]

Right, I saw that and hence the post here. However, on this machine, which is an ordinary GCP machine with 4 L4s, it did not work.
Given the point about lists, should I expect a line like "0,1" to work for selecting GPUs 0 and 1?

[ethanhe42]

how did you launch the job?
did you use torchrun or https://github.com/NVIDIA/NeMo-Megatron-Launcher