GPU utilization

[SeifeddineGharbi]

I wanted to test the search with the test search script. Since GPUs 0 and 1 are being used for another process, I ran this command:

CUDA_VISIBLE_DEVICES=2,3 bash scripts/test_search.sh

Even though the following log status shows in the resssources requested 1.96/2 GPUs which makes perfect sense and every model is taking 2 CPUs and 0.49 GPU
== Status ==
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 8.0/20 CPUs, 1.96/2 GPUs
Memory usage on this node: 91.1/269.9 GB
Result logdir: /home/sgharbi/pba/results/test_search
Number of trials: 4 ({'RUNNING': 4})
RUNNING trials:

• RayModel_0: RUNNING, [2.0 CPUs, 0.49 GPUs], [pid=425228], 87 s, 1 iter
• RayModel_1: RUNNING, [2.0 CPUs, 0.49 GPUs], [pid=425227], 87 s, 1 iter
• RayModel_2: RUNNING, [2.0 CPUs, 0.49 GPUs], [pid=425241], 87 s, 1 iter
• RayModel_3: RUNNING, [2.0 CPUs, 0.49 GPUs], [pid=425233], 173 s, 2 iter

But when i look with nvidia-smi there no usage of GPUs 2 and 3!

Am I doing something wrong here please? If so, how to make it really utilize the GPUs, thank you so much!

[SeifeddineGharbi]

@arcelien Can you look into my issue please? Thank you!