TL/MLX5: fix fences in a2a's WQEs

[samnordmann]

[Edited]

What

Fix fences in WQEs.

QPs used by each node leader:

• a QP with loop-back connection, which we call "UMR QP"
• one QP per remote peer (+1 with loop-back connection), which we call "RDMA QP"

Here the different WQEs in the algorithm:

1. UMR WQE posted to "UMR QP". CPU then blocks until completion of this WQE.
2. Transpose WQE+ RDMA write WQE + atomic fetch_and_add WQE, all posted to "RDMA QP"
3. wait-on-data posted to "UMR QP"

Conclusion regarding flags:

• UMR WQE doesn't need any fence since it is the first WQE posted.
• The transpose WQE, which consumes UMR, needs a small fence. doesn't need any fence.
• RDMA write, which consumes transpose, needs a small fence because it only uses local data
• atomic fetch_and_add WQE needs a strong fence because it signals completion of RDMA write to the remote peer doesn't need any fence
• Wait-on-data doesn't need any fence.

[raminudelman]

Regarding:

UMR WQE doesn't need any fence since it is the first WQE posted

Do we have a single UMR per AllToAll operation? If the answer is yes - then I agree, no need for any fence (assuming we also have a barrier and the UMR does not change/modify any Mkey that maps memory that might be accessed by previous operations).

[samnordmann]

Regarding:

UMR WQE doesn't need any fence since it is the first WQE posted

Do we have a single UMR per AllToAll operation? If the answer is yes - then I agree, no need for any fence (assuming we also have a barrier and the UMR does not any memory that might be accessed by previous operations).

We have two UMRs per AllToAll, on for the send and one for the recv key. Indeed the src and recv buffers are assumed to be ready when the collective is called, and UMRs are the first WQEs

[raminudelman]

It hangs because if you don't set CE to be 0x2 (as specified by MLX5_WQE_CTRL_CQ_UPDATE) - you won't get a CQE for this WQE you're posting...

[samnordmann]

Regarding:

UMR WQE doesn't need any fence since it is the first WQE posted

Do we have a single UMR per AllToAll operation? If the answer is yes - then I agree, no need for any fence (assuming we also have a barrier and the UMR does not any memory that might be accessed by previous operations).

We have two UMRs per AllToAll, on for the send and one for the recv key. Indeed the src and recv buffers are assumed to be ready when the collective is called, and UMRs are the first WQEs

@raminudelman

Sorry my previous PR description and analysis was misleading. I edited the description and changed further the flags, please see the last commit.

[raminudelman]

I'm not sure why:

wait-on-data posted to "UMR QP"

Since IIUC it can also be posted on an "RDMA QP" but I don't think it matters.

Another note:
A clear optimization space that is left here on the table is the fact that the CPU blocks until the UMR WQE is completed. This latency is exposed (from UCC's AllToAll perspective...). I would want to see a breakdown of how much latency we expose here in different systems scales. If it's <5% of the latency of the AllToAll probably there is not need to try to optimize it but if it's more I think it might be worthwhile. I expect it to be obviously exposed in a smaller scale rather than in large scale systems.

[raminudelman]

Regarding

atomic fetch_and_add WQE needs a strong fence because it signals completion of RDMA write to the remote peer

IIRC, InfiniBand's ordering semantics guarantee that Atomic operations are executed in order (according to the message/PSN ordering) on the responder side. So, no need to indicate "fence" in atomic operation on the requestor side. @samnordmann, please double check this.

[samnordmann]

I'm not sure why:

wait-on-data posted to "UMR QP"

Since IIUC it can also be posted on an "RDMA QP" but I don't think it matters.

Right, it's just implemented this way, but this doesn't need to be like that. In principle, it also allows Wait-on-Data and RDMA to be processed in parallel, even though I doubt it brings a concrete benefit.

Another note:
A clear optimization space that is left here on the table is the fact that the CPU blocks until the UMR WQE is completed. This latency is exposed (from UCC's AllToAll perspective...). I would want to see a breakdown of how much latency we expose here in different systems scales. If it's <5% of the latency of the AllToAll probably there is not need to try to optimize it but if it's more I think it might be worthwhile. I expect it to be obviously exposed in a smaller scale rather than in large scale systems.

If we want to avoid blocking CPU, it means we rely on the QP ordering. So IIUC what you suggests is relevant only for WQE posted on UMR QP, which has a loopback connection (does it need to loop-back? need to double-check). So the only possible WQE we can post are the local Transpose+LDMA, right?

Besides, when reusing the same src/dst buffer, the registration is cached and so UMR is amortized accross iterations. So UMR does not represent a significant latency in that case.

IIRC, InfiniBand's ordering semantics guarantee that Atomic operations are executed in order (according to the message/PSN ordering) on the responder side. So, no need to indicate "fence" in atomic operation on the requestor side. @samnordmann, please double check this.

got it, thanks!

[manjugv]

@raminudelman is this good from your perspective?

[raminudelman]

@manjugv ,

I had a talk with @samnordmann last week and this PR is indeed good and fixes some inefficiency. There is still a space for optimizations here but before implementing them (which might require much more effort), we agreed with @samnordmann that a deeper profiling of the current performance and bottlenecks is required to really be able to know where optimizations should be applied.