Inquiry About UCX GPU Memory Usage

[supunkamburugamuve]

Hi Everyone,

We are using UCX to send messages between NVIDIA GPUs over InfiniBand. We employ cudaAsync memory and have configured UCX with the following settings:

• UCX_RNDV_FRAG_SIZE: host: 512K, cuda: 1024K
• UCX_RNDV_FRAG_ALLOC_COUNT: 128
• UCX_RNDV_FRAG_MEM_TYPE: 'cuda'

My understanding is that UCX is using up to 128MB of GPU memory for bounce buffers with the above configuration.

Is it possible for UCX to allocate more than 128MB in this configuration?

Thank you,
Supun

[brminich]

Yes, it is possible. UCX uses a memory pool to manage bounce buffers, and this value determines the number of elements allocated per chunk. If there are not enough elements available for an operation, UCX may allocate an additional chunk.

[supunkamburugamuve]

Thank you. Is there a way to limit the size of the memory pool?

[brminich]

currently it is not possible without modifying the code, we are working on limiting it

[PeterGottesman]

Hi Mikhail, I am working with Supun and have looked into this recently. I am wondering if I am correct to think we can adjust the UCX_CUDA_COPY_MAX_EVENTS option to limit the maximum amount of memory used? Would using that result in a failure if the limit is reached?

[brminich]

yes, it will result in a failure

[supunkamburugamuve]

Thanks for the clarifications. We are working towards a workaround to limit the usage of the pool until the feature is available. Is there a timeline for this feature?