Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #30

artpol84 · 2025-03-15T04:28:16Z

No description provided.

Isolate header files from compile-time dependencies (may not be required if global config.h is distributed along with header files) Signed-off-by: Artem Y. Polyakov <[email protected]>

artpol84 · 2025-03-15T04:28:55Z

Backend Unit-test

[artemp@login01 test]$ ./ucx_mo_backend_test


****************************************************
    Inter-agent memory transfer test P-Thr=OFF
         (DRAM -> DRAM)
****************************************************


Synchronous handshake complete

READ test (10) iterations
        READ
                NOTE: Testing non-inline Transfer path!
                Data verification: OK
...
        READ
                NOTE: Testing non-inline Transfer path!
                Data verification: OK

WRITE test (10) iterations
        WRITE
                NOTE: Testing non-inline Transfer path!
                Data verification: OK
...
        WRITE
                NOTE: Testing non-inline Transfer path!
                Data verification: OK

READ/NOTIF test (10) iterations
        READ/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK
...
        READ/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK

WRITE/NOTIF test (10) iterations
        WRITE/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK
...
        WRITE/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK

Test genNotif operation
         gnNotif to Agent2
                Checking notification flow: OK
...
         gnNotif to Agent2
                Checking notification flow: OK


****************************************************
    Inter-agent memory transfer test P-Thr=ON
         (DRAM -> DRAM)
****************************************************


Synchronous handshake complete

READ test (10) iterations
        READ
                NOTE: Testing non-inline Transfer path!
                Data verification: OK
...
        READ
                NOTE: Testing non-inline Transfer path!
                Data verification: OK

WRITE test (10) iterations
        WRITE
                NOTE: Testing non-inline Transfer path!
                Data verification: OK
...
        WRITE
                NOTE: Testing non-inline Transfer path!
                Data verification: OK

READ/NOTIF test (10) iterations
        READ/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK
...
        READ/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK

WRITE/NOTIF test (10) iterations
        WRITE/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK
...
        WRITE/NOTIF
                NOTE: Testing non-inline Transfer path!
                Checking notification flow: OK
                Data verification: OK

Test genNotif operation
         gnNotif to Agent2
                Checking notification flow: OK

...
         gnNotif to Agent2
                Checking notification flow: OK

test/unit/plugins/ucx_mo/ucx_mo_backend_test.cpp

src/plugins/ucx_mo/ucx_mo_backend.cpp

src/plugins/ucx_mo/meson.build

src/plugins/ucx_mo/ucx_mo_backend.cpp

src/plugins/ucx_mo/ucx_mo_plugin.cpp

Add new UCX-based backend that allows associating NIXL logical "devices" with different UCX workers. The primary motivation is that UCX v1.18 doesn't support more than one GPGPU per UCX context. NOTE: this is expected to be fixed in UCX v1.19 so this backend might be viewed as a workaround unless other uses will be found. Signed-off-by: Artem Y. Polyakov <[email protected]>

artpol84 · 2025-03-15T05:08:03Z

nixl example (hacked)

diff --git a/test/nixl/agent_example.cpp b/test/nixl/agent_example.cpp
index c450814..59cd290 100644
--- a/test/nixl/agent_example.cpp
+++ b/test/nixl/agent_example.cpp
@@ -368,8 +368,8 @@ int main()
     printParams(init2);

     nixlBackendH* ucx1, *ucx2;
-    ret1 = A1.createBackend("UCX", init1, ucx1);
-    ret2 = A2.createBackend("UCX", init2, ucx2);
+    ret1 = A1.createBackend("UCX_MO", init1, ucx1);
+    ret2 = A2.createBackend("UCX_MO", init2, ucx2);

     assert (ret1 == NIXL_SUCCESS);
     assert (ret2 == NIXL_SUCCESS);

[artemp@login01 nixl]$ ./agent_example
Using plugin file: /global/home/users/artemp/nixl/nixl/build0/pluginlist
Found plugin: UCX at path: /global/home/users/artemp/nixl/nixl/build0/src/plugins/ucx/libplugin_UCX.so
Found plugin: UCX_MO at path: /global/home/users/artemp/nixl/nixl/build0/src/plugins/ucx/libplugin_UCX.so
Successfully loaded plugin 'UCX' version 1.0.0 from /global/home/users/artemp/nixl/nixl/build0/src/plugins/ucx/libplugin_UCX.so
Successfully loaded plugin 'UCX_MO' version 1.0.0 from /global/home/users/artemp/nixl/nixl/build0/src/plugins/ucx/libplugin_UCX.so
Registering built-in static plugins...
Available plugins:
UCX
UCX_MO
Params before init:
Parameters:
  ucx_devices =
Parameters:
  ucx_devices =
Plugin already loaded: UCX_MO
Plugin already loaded: UCX_MO
Params after init:
Parameters:
  ucx_devices =
Parameters:
  ucx_devices =
Agent1's Metadata: nixlSerDes|AgenAgent001|Conn|tUCX_MO|cp ����=\!&
        ��Q8�S�LY��6#��w!n�m
?�����Q8S�>K��77#��!&�
                      ��Q8Wt�NTv�6#����|
MemSection|nixlSecElm|bkndUCX_MO|nixlDList
nixlSDList|t|u|s||$`l�|$�n�|
Agent2's Metadata: nixlSerDes|AgenAgent002|Conn|tUCX_MO|cp �#@8�,!&
        ��Q8�S�LY��6#��!n�m
?�����Q8S�>K��77#���!&�
                       ��Q8Wt�NTv�6#��k��|
MemSection|nixlSecElm|bkndUCX_MO|nixlDList
nixlSDList|t|u|s||$pm�|
Transfer request from 0x9a6c60 to 0x9a6d70
Transfer was posted
Transfer verified
performing sideXferTest with backends 0x978910 0x9a27a0
Starting sideXferTest
Got backend
perf setup done
prepXferSide, total time for 10 iters: 1s 995447us
time per 2 preps 199545us
 src 0 0x9d3990
 dst 0 0x9d31e0
 src 1 0x9d4940
 dst 1 0x9d4d50
 src 2 0x9d5160
 dst 2 0x9ed2d0
 src 3 0x9ed6e0
 dst 3 0x9edaf0
Ready to prepare side
prep done, starting transfers
transfer 1 done
transfer 2 done
transfer 3 done
Performing local test
Local transfer was posted
Test done

aranadive · 2025-03-15T06:10:52Z

src/plugins/ucx_mo/ucx_mo_plugin.cpp

+     get_plugin_version,
+     get_backend_options
+ };
+ #ifdef STATIC_PLUGIN_UCX


If plugin name is UCX_MO the flag here should be STATIC_PLUGIN_UCX_MO

mkhazraee · 2025-03-19T00:29:41Z

Recreated in PR 58.

backend/ucx: Move CUDA declarations to cpp-file

210ba6e

Isolate header files from compile-time dependencies (may not be required if global config.h is distributed along with header files) Signed-off-by: Artem Y. Polyakov <[email protected]>

artpol84 requested review from tstamler, mkhazraee and aranadive March 15, 2025 04:28

artpol84 had a problem deploying to GITLAB March 15, 2025 04:29 — with GitHub Actions Failure

artpol84 force-pushed the topic/multi-object/final branch from 3d52724 to a8146f0 Compare March 15, 2025 04:31

artpol84 had a problem deploying to GITLAB March 15, 2025 04:32 — with GitHub Actions Failure

artpol84 commented Mar 15, 2025

View reviewed changes

artpol84 had a problem deploying to GITLAB March 15, 2025 04:45 — with GitHub Actions Failure

artpol84 force-pushed the topic/multi-object/final branch from 7f9314f to 0f7e51e Compare March 15, 2025 04:52

artpol84 had a problem deploying to GITLAB March 15, 2025 04:52 — with GitHub Actions Failure

artpol84 had a problem deploying to GITLAB March 15, 2025 04:53 — with GitHub Actions Failure

artpol84 had a problem deploying to GITLAB March 15, 2025 04:54 — with GitHub Actions Failure

artpol84 force-pushed the topic/multi-object/final branch from 38f7017 to 033a15a Compare March 15, 2025 05:01

artpol84 had a problem deploying to GITLAB March 15, 2025 05:01 — with GitHub Actions Failure

artpol84 force-pushed the topic/multi-object/final branch from 033a15a to 04b2c0b Compare March 15, 2025 05:04

artpol84 had a problem deploying to GITLAB March 15, 2025 05:04 — with GitHub Actions Failure

aranadive reviewed Mar 15, 2025

View reviewed changes

artpol84 had a problem deploying to GITLAB March 16, 2025 02:40 — with GitHub Actions Failure

artpol84 had a problem deploying to GITLAB March 18, 2025 20:03 — with GitHub Actions Failure

artpol84 temporarily deployed to GITLAB March 19, 2025 00:09 — with GitHub Actions Inactive

artpol84 temporarily deployed to GITLAB March 19, 2025 00:10 — with GitHub Actions Inactive

artpol84 mentioned this pull request Mar 19, 2025

Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #58

Merged

mkhazraee closed this Mar 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #30

Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #30

artpol84 commented Mar 15, 2025

artpol84 commented Mar 15, 2025

artpol84 commented Mar 15, 2025

aranadive Mar 15, 2025

mkhazraee commented Mar 19, 2025

Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #30

Multi-Object (MO) UCX backend implementation (rebased over code restructuring PR) #30

Conversation

artpol84 commented Mar 15, 2025

artpol84 commented Mar 15, 2025

artpol84 commented Mar 15, 2025

aranadive Mar 15, 2025

Choose a reason for hiding this comment

mkhazraee commented Mar 19, 2025