UCP/RMA/FLUSH: Dynamic Selection of Strong vs. Weak Fence

[michal-shalev]

What?

Ensure Strong Fence is used only in scenarios where it is genuinely required.

• Add unflushed_lanes to EP struct
• Add fence_seq to EP and worker structs
• Add UCP_FENCE_MODE_EP_BASED fence mode
• Implement per-EP fences
• Handle fence during one-sided operations (instead of worker fence)
• Add a correctness test for EP-based fence mode

Why?

The current implementation of ucp_worker_fence always uses a Strong fence regardless of whether multiple lanes were used for operations.
This inefficiency occurs because the system lacks runtime information on which lanes were used.
This leads to suboptimal performance in Single-Rail scenarios, even though Weak Fence would suffice.

How?

To dynamically select between Strong and Weak Fence modes, this PR introduces a mechanism that tracks lane usage at runtime and applies the appropriate fencing at the EP level.

[brminich]

can we also avoid fence if all previously issued operations are already completed?

[yosefe]

can we pass rbuf.ptr() also as void*?

[michal-shalev]

AFAIU rbuf.ptr() cannot be passed as void* for remote_addr because ucp_put_nbx() explicitly requires a uint64_t remote address.

[yosefe]

i guess perform_nbx_with_fence could convert void* to uint64_t because it's calling ucp_put_nbx directly

[yosefe]

how does the assembly compare? seems more complicated IMO

[michal-shalev]

I used Godbolt to check:

#define FLAG 0x8
#define LANE 0x6

void update_with_branch(uint64_t *value, uint64_t flags) {
    if (flags & FLAG) {
        *value |= LANE;
    }
}

void update_branchless(uint64_t *value, uint64_t flags) {
    *value |= LANE & -!!(flags & FLAG);
}

update_with_branch:
        and     esi, 8
        je      .L1
        or      QWORD PTR [rdi], 6
.L1:
        rep ret
update_branchless:
        sal     rsi, 60
        sar     rsi, 63
        and     esi, 6
        or      QWORD PTR [rdi], rsi
        ret

This shows that update_with_branch includes a conditional branch (je .L1),
while update_branchless avoids branching entirely by using bitwise operations.

[yosefe]

but actually we have 2 flags, can you check in godbolt something like

#define FLAG1 0x8
#define FLAG2 0x20

void update_with_branch(uint64_t *value, uint64_t flags) {
    if (flags & FLAG1) {
        *value |= FLAG2;
    }
}

void update_branchless(uint64_t *value, uint64_t flags) {
    *value |= FLAG1 & -!!(flags & FLAG2);
}

[michal-shalev]

I've updated my original comment @yosefe

[yosefe]

BTW, maybe we could update unflushed_lanes unconditionally?

[michal-shalev]

I think it's the best solution so far, pushed another commit

[yosefe]

i guess perform_nbx_with_fence could convert void* to uint64_t because it's calling ucp_put_nbx directly

[gleon99]

Already approved + minor CI fix.
@michal-shalev please squash.