verbs: Query QP data in order on non x86 platforms

[dkkranz]

EFA can support 128 bytes blocks write in-order for some Grace platforms. Move the check on x86 architecture to the mlx5 provider implementation.

Reviewed-by: Michael Margolin [email protected]
Reviewed-by: Yonatan Nachum [email protected]

[jgunthorpe]

I don't think this is right, it would break things like PPC. The platform check is supposed to be in the common code, and the driver check is only about how the device works on PCI.

You should probably just add ARM to the inclusion list, though I imagine there are some wonky ARMs that don't work, they may not matter in practice.

[dkkranz]

Applied your suggestion, thanks

[dkkranz]

@jgunthorpe Kind reminder

[jgunthorpe]

I don't know.. I just feel uneasy about this. ARM is such a wide and varied architecture I don't know if we can really say it is actually reliably in order. Grace might be just because it has 128 byte lines, but that doesn't mean a different ARM chip will be. This could break some of the MPIs running on ARM supercomputers.. Do you think you could limit it just to Grace? Did someone from NVIDIA confirm that this is actually true on Grace?

[jgunthorpe]

I consulted with people here and unfortunately I don't think using ARM platform as a condition is sufficient. Even looking at NVIDIA CPUs this only works on certain CPUs, with certain configurations.

I guess EFA could be special since it really only exists in AWS instances and you guys will do the validation. I'd still strongly recommend that you query your device FW to see if it is supported as I'm not sure you will find all future systems will work this way.

So maybe the right thing is to add a new op query_qp_data_in_order_force() or something like that that skips the architecture check in the core code. I don't really want to have drivers doing architecture checks..

[dkkranz]

@jgunthorpe Please review the last revision that adds a flag to skip the architecture check.

[jgunthorpe]

I didn't mean a flag to userspace, there is nothing userspace can do with this. I ment some kind of flag from EFA to the core code to disable the arch checks in the core code just for EFA.

[mrgolin]

I don't really want to have drivers doing architecture checks..

But this is exactly what you are suggesting now, and I'm not sure why we need to have architecture checks in libibverbs either.

I think there is use for a common way to query whether data is written in order from device perspective only, for instance data polling from GPU isn't necessarily affected by the CPU architecture. Don't you agree?