Regarding the issue of GPU communication

[EmmaLee-c]

May I ask what the function of parameter UCX_IB_GPU_DIRECT_RDMA is? In the UCX code, I only found the processing logic for UCX_IB_GPU_DIRECT_RDMA!=0
In uct_ib_md_open_common:
if (md_config->enable_gpudirect_rdma != UCS_NO) {
/* check if GDR driver is loaded /
uct_ib_check_gpudirect_driver(md, "/sys/kernel/mm/memory_peers/nv_mem/version", UCS_MEMORY_TYPE_CUDA);
/ check if ROCM KFD driver is loaded /
uct_ib_check_gpudirect_driver(md, "/dev/kfd", UCS_MEMORY_TYPE_ROCM);
/ Check for dma-buf support */
uct_ib_md_check_dmabuf(md);
}

I have conducted some experiments and found that setting UCX_IB_GPU_DIRECT_RDMA=0 does indeed cause the network card to no longer directly manipulate GPU memory data, but instead copy it to the CPU through cuda_copy before executing the RDMA operation.

May I ask where else can the ucx source code determine UCX_IB_GPU_DIRECT_RDMA=0, to decide whether to use GPU Direct RDMA?
Thank you.

[yosefe]

the default behavior is to enable GPU direct if nv_peer_memory driver is loaded, unless UCX_IB_GPU_DIRECT_RDMA is set to 0

[drmichaeltcvx]

Using the latest HPC-X on systems running Alma8 (RHEL8-like Linux, with MLNX_OFED_LINUX-24.04-0.7.0.0) what kernel modules need to be loaded before UCX can utilize the most efficient (copy avoiding) communications algorithms for inter-GPU (intra-node and inter-node) communications? Documentation is somehow unclear.

Is nv_peer_memory necessary?
Is GDRcopy necessary?

Does xpmem enable faster intranode communications for either inter-GPU or or inter process communications ? Is it mandatory ?

Thanks!

[yosefe]

Is nv_peer_memory necessary? - it's not needed on newest kernels that support dmabuf
Is GDRcopy necessary? - it helps small messages performance but not so much for large messages
xpmem allows fast intra-node host-host memory communications, it is supplied as part of MLNX_OFED/DOCA_HOST, it's not mandatory and UCX would use the built-in process_vm_readv/writev if not supported.