Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TL/MLX5: fix fences in a2a's WQEs #1069

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

TL/MLX5: fix fences in a2a's WQEs #1069

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6e1ec37
TL/MLX5: fix fences in WQEs
samnordmann Jan 9, 2025
a84a08a
CODESTYLE: align
samnordmann Jan 9, 2025
c226e9f
TL/MLX5: further WQE flags revision
samnordmann Jan 10, 2025
7f9f604
TL/MLX5: remove unnecessary fence on atomic WQE
samnordmann Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 14 additions & 14 deletions src/components/tl/mlx5/tl_mlx5_wqe.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ ucc_status_t ucc_tl_mlx5_post_transpose(struct ibv_qp *qp, uint32_t src_mr_lkey,
uint32_t n_ds = 4;
struct ibv_qp_ex * qp_ex = ibv_qp_to_qp_ex(qp);
struct mlx5dv_qp_ex * mqp = mlx5dv_qp_ex_from_ibv_qp_ex(qp_ex);
int fm_ce_se = 0;
int fm_ce_se = MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE;
char wqe_desc[n_ds * DS_SIZE];
struct mlx5_wqe_ctrl_seg *ctrl;
struct mlx5_wqe_data_seg *data;
Expand Down Expand Up @@ -153,14 +153,14 @@ ucc_status_t ucc_tl_mlx5_post_umr(struct ibv_qp * qp,
sizeof(struct mlx5_wqe_mkey_context_seg) +
sizeof(struct mlx5_wqe_umr_pointer_seg)) /
DS_SIZE;
uint8_t fm_ce_se =
MLX5_WQE_CTRL_INITIATOR_SMALL_FENCE | MLX5_WQE_CTRL_CQ_UPDATE;
struct ibv_qp_ex * qp_ex = ibv_qp_to_qp_ex(qp);
struct mlx5dv_qp_ex * mqp = mlx5dv_qp_ex_from_ibv_qp_ex(qp_ex);
struct mlx5_wqe_ctrl_seg * ctrl;
struct mlx5_wqe_umr_ctrl_seg * umr_ctrl_seg;
uint8_t fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It hangs because if you don't set CE to be 0x2 (as specified by MLX5_WQE_CTRL_CQ_UPDATE) - you won't get a CQE for this WQE you're posting...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right! Thx. I was missing this

ucc/src/components/tl/mlx5/alltoall/alltoall_mkeys.c

Line 77 in 0c0fc21

while (!ret) {

Copy link
Collaborator Author

@samnordmann samnordmann Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the wrong impression that UMR and transpose were posted on the same QP, and ordered through a fence. But they are actually posted on two different QPs, and the CPU blocks for UMR completion before posting the Transpose.

struct ibv_qp_ex *qp_ex = ibv_qp_to_qp_ex(qp);
struct mlx5dv_qp_ex *mqp =
mlx5dv_qp_ex_from_ibv_qp_ex(qp_ex);
struct mlx5_wqe_ctrl_seg *ctrl;
struct mlx5_wqe_umr_ctrl_seg *umr_ctrl_seg;
struct mlx5_wqe_mkey_context_seg *mk_seg;
struct mlx5_wqe_umr_pointer_seg * pseg;
struct mlx5_wqe_umr_pointer_seg *pseg;
char wqe_desc[n_ds * DS_SIZE];
int xlat_size;

Expand Down Expand Up @@ -270,12 +270,12 @@ ucc_status_t ucc_tl_mlx5_post_wait_on_data(struct ibv_qp *qp, uint64_t value,
void *task_ptr)
{

uint32_t opcode = MLX5_OPCODE_WAIT;
uint32_t opmode = 0x1; //wait on data
uint32_t n_ds = 3; //CTRL + Wait on Data of Size 2
struct ibv_qp_ex * qp_ex = ibv_qp_to_qp_ex(qp);
struct mlx5dv_qp_ex *mqp = mlx5dv_qp_ex_from_ibv_qp_ex(qp_ex);
uint8_t fm_ce_se = MLX5_WQE_CTRL_FENCE | MLX5_WQE_CTRL_CQ_UPDATE;
uint32_t opcode = MLX5_OPCODE_WAIT;
uint32_t opmode = 0x1; //wait on data
uint32_t n_ds = 3; //CTRL + Wait on Data of Size 2
struct ibv_qp_ex *qp_ex = ibv_qp_to_qp_ex(qp);
struct mlx5dv_qp_ex *mqp = mlx5dv_qp_ex_from_ibv_qp_ex(qp_ex);
uint8_t fm_ce_se = MLX5_WQE_CTRL_CQ_UPDATE;
char wqe_desc[n_ds * DS_SIZE];
struct mlx5_wqe_ctrl_seg *ctrl;
wait_on_data_seg_t * wseg;
Expand Down
Loading