Instances‘ meaning

[Eevan-zq]

Hello Sir, I would like to ask about the instance parameter that appears when defining a custom algorithm in MSCCL-TOOLS. However, I have always been unclear about its meaning.

When looking at your code, I noticed the statement for inst in range(ninstance): in the to_xml.py file. I suspect that ninstance represents the number of processes created for each GPU. I am not sure if my understanding is correct.

[liangyuRain]

The ninstance represents the number of identical communication algorithms you are simultaneously running. In GPU hardware, often you need multiple instances sending/receiving data simultaneously to fully utilize bandwidth.

[Eevan-zq]

Thank you for your response. I would like to follow up by asking: what factors determine the number of instances ? For example, is it proportional to the number of threads per GPU, the number of channels, or something else?

[liangyuRain]

Usually, if you aim for throughput (for large data sizes), you want as many instances as possible. If the data size is small, then you dont want a lot of instances. In practice, different data sizes have different optimal ninstance. People usually switch schedules based on data size.

[Eevan-zq]

Yes, in my understanding, an instance is equivalent to the concept of parallelism, where multiple parallel GPUs execute the collective communication primitive algorithm simultaneously, right?

[liangyuRain]

The number of GPUs in collective communication is constant regardless of ninstance. Increasing ninstance is increasing the number of parallel threadblocks or SMs executing the schedule.

[Eevan-zq]

Thank you for your response. So, in the XML file called by MSCCL, under each GPU tag, there are multiple thread tags, and the value of ninstance is positively correlated with the number of these thread blocks.