Provide a way to access the GRES_IDX attribute for a job

[chrissamuel]

Details

• Slurm Version: 17.11.7
• Python Version: 3.4 (RHEL7)
• Cython Version: ??
• PySlurm Branch: 17.11.0
• Linux Distribution: RHEL7

Issue

It would be really useful to us for our job monitoring program to be able to get the GRES_IDX information from running jobs so we can highlight the GPU the job is using for our users so they can see how much utilisation it is getting.

Our issue for this is: conradtchan/jobmon#1

This looks like GRES_IDX=gpu(IDX:0) or GRES_IDX=gpu(IDX:1) or GRES_IDX=gpu(IDX:0-1) for our systems with dual GPUs. I'm not sure what it would look like for a system with say 4 GPUs if the allocation is not contiguous.

Thanks for considering this!

All the best,
Chris

[giovtorres]

Thanks. I was hoping this one would be easy. Here is the code that needs to get wrapped: https://github.com/SchedMD/slurm/blob/5073024350eb79c8c5a9964e800bc0ce3ab93d59/src/api/job_info.c#L706-L803.

That may take me some time.

How would you like the output of this attribute, as one string that matches the scontrol output, or a list of strings if there are more than one?

[chrissamuel]

Sorry for not spotting this before, I've just passed your query on to the developer!

[chrissamuel]

Got a reply from the developer today, he said:

Sorry for the late reply... a list of strings would be great!

Hope that helps!
Chris

[giovtorres]

Hi @chrissamuel. I know it's been a while, but I think I got the code wrapped to get GRES_IDX in one of the 18.08 branches I'm working on. I should be able to backport to an older branch. Are you still on Slurm 17.11.7?

[chrissamuel]

Hi @giovtorres!

Swinburne is on 18.08.x now, but I've since moved to the US for love and for work and am now at NERSC (still doing HPC). But I'll still see updates here and let them know.

Thanks for this!
Chris

[giovtorres]

The gres_idx branch should work with a later version of Cython. I'll merge it into master soon after I figure out why it is failing on older Cython versions.

[chrissamuel]

Thanks! Passed that back to them.