UCT/CUDA: Use sys_dev to allocate CUDA memory #10553

tvegas1 · 2025-03-14T15:13:12Z

What?

Re-adds existing API PR #10448 with latest comments addressed, and UCT implementation.

Why?

Fragments used for pipeline need to use the same GPU as the user GPU.

How?

Set context if needed, if not available find first usable primary context. If explicit context was requested and not found, return an error.

tvegas1 · 2025-03-14T15:19:14Z

@rakhmets, we can probably eventually reuse here the ctx retain cache from #10545 when it's merged?

rakhmets · 2025-03-14T15:22:25Z

@rakhmets, we can probably eventually reuse here the ctx retain cache from #10545 when it's merged?

Yes, exactly.

rakhmets · 2025-03-14T15:34:48Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+        return UCS_OK;
+    }
+
+    status = UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRetain(&primary_ctx, device),


I think it is better to check the primary device context using cuDevicePrimaryCtxGetState. The main use case that should be covered when user allocates memory on the device I, then switch to device J, and we should allocate staging buffer on device I. Thus, if cuDevicePrimaryCtxGetState returns inactive state for the given device, then we should report an error instead of retaining and activating primary context on the device without active primary context.

added function, and log

@rakhmets why is it considered an error?

i assume this code will be modified to use common func for context retainig, so no need to review it now, right?

removed the two function above and using uct_cuda_primary_ctx_retain()

src/uct/cuda/cuda_copy/cuda_copy_md.c

brminich · 2025-03-14T16:20:53Z

src/uct/cuda/base/cuda_md.c

@@ -61,6 +61,34 @@ void uct_cuda_base_get_sys_dev(CUdevice cuda_device,
    *sys_dev_p = UCS_SYS_DEVICE_ID_UNKNOWN;
 }

+static CUdevice sys_dev_to_device[UCS_SYS_DEVICE_ID_MAX] = {
+    [0 ... UCS_SYS_DEVICE_ID_MAX - 1] = CU_DEVICE_INVALID


isn't it gcc extension, but not a C99?

brminich · 2025-03-14T16:43:21Z

test/gtest/uct/cuda/test_switch_cuda_device.cc


 class test_switch_cuda_device : public test_md {
-public:
+protected:
+    int num_devices;


minor:

Suggested change

int num_devices;

int m_num_devices;

brminich · 2025-03-14T16:43:44Z

test/gtest/uct/cuda/test_switch_cuda_device.cc

 };

 template<class T>
 void test_switch_cuda_device::detect_mem_type(ucs_memory_type_t mem_type) const
 {
-    int num_devices;
-    ASSERT_EQ(cudaGetDeviceCount(&num_devices), cudaSuccess);
+    int current_device;


no need to move

this is now done as part of init(), unless I miss something?

i meant no need to move initialization of int current_device;

brminich · 2025-03-14T16:52:22Z

src/uct/cuda/base/cuda_md.c

@@ -90,6 +118,8 @@ uct_cuda_base_query_md_resources(uct_component_t *component,
            continue;
        }

+        sys_dev_to_device[sys_dev] = cuda_device;


shall we populate this array in uct_cuda_base_get_sys_dev instead?
it is not expected that user will query md tl resources for every cudaDevice visible for the thread/process

The user does not need to query all devices, but indeed, does need to query at least one device to trigger this cache population.

Moved to uct_cuda_base_get_sys_dev() for now but since it adds if(){} check, maybe we could move it to ctor function?

ok to have it here then

brminich · 2025-03-17T16:08:58Z

src/uct/api/uct.h

@@ -1712,6 +1712,7 @@ typedef struct uct_allocated_memory {
    ucs_memory_type_t        mem_type; /**< type of allocated memory */
    uct_md_h                 md;       /**< if method==MD: MD used to allocate the memory */
    uct_mem_h                memh;     /**< if method==MD: MD memory handle */
+    ucs_sys_device_t         sys_dev;  /**< System device for allocated memory */


btw, seems like we use sys_device more often in the APIs, maybe better to use it here as well

brminich · 2025-03-17T16:11:26Z

src/uct/cuda/base/cuda_md.c

@@ -90,6 +118,8 @@ uct_cuda_base_query_md_resources(uct_component_t *component,
            continue;
        }

+        sys_dev_to_device[sys_dev] = cuda_device;


ok to have it here then

brminich · 2025-03-17T16:12:09Z

test/gtest/uct/cuda/test_switch_cuda_device.cc

 };

 template<class T>
 void test_switch_cuda_device::detect_mem_type(ucs_memory_type_t mem_type) const
 {
-    int num_devices;
-    ASSERT_EQ(cudaGetDeviceCount(&num_devices), cudaSuccess);
+    int current_device;


i meant no need to move initialization of int current_device;

tvegas1 · 2025-03-17T19:33:06Z

Repro failures with container nvidia/cuda:11.8.0-devel-ubuntu20.04 and nvidia-utils-555, some comments remain to be addressed.

src/uct/cuda/cuda_copy/cuda_copy_md.c

src/uct/api/uct.h

src/uct/base/uct_mem.c

src/uct/cuda/cuda_copy/cuda_copy_md.c

yosefe · 2025-03-19T08:08:10Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+        return UCS_OK;
+    }
+
+    status = UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRetain(&primary_ctx, device),


@rakhmets why is it considered an error?

src/uct/cuda/cuda_copy/cuda_copy_md.c

tvegas1 · 2025-03-19T16:02:55Z

all comments addressed, skip inactive disabled for now to see if any failure.

yosefe · 2025-03-19T16:08:18Z

src/uct/base/uct_mem.c

+            if ((mem_type != UCS_MEMORY_TYPE_HOST) ||
+                (sys_dev != UCS_SYS_DEVICE_ID_UNKNOWN)) {


add helper inline function, smth like
uct_mem_is_host_alloc(params)

src/ucs/sys/topo/base/topo.c

src/uct/cuda/base/cuda_md.c

src/uct/cuda/cuda_copy/cuda_copy_md.c

yosefe · 2025-03-19T16:18:34Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+    status = UCT_CUDADRV_FUNC(cuDevicePrimaryCtxGetState(device, &flags,
+                                                         &active),
                              log_level);
    if (status != UCS_OK) {
        return status;
-    } else if (!active) {
+    } else if (!active && md->config.skip_inactive_primary_ctx) {


to rebase on Raul's PR for VMM detection

src/uct/cuda/cuda_copy/cuda_copy_md.c

test/gtest/uct/cuda/test_switch_cuda_device.cc

yosefe · 2025-03-19T17:09:46Z

==== Running ucx_info env vars test ====
+ UCX_IB_PORTS=mlx5_0:1
+ ./src/tools/info/ucx_info -epw -u t
+ grep -q -E UCX_IB_PORTS
+ grep unused
[swx-rdmz-ucx-gpu-02:64825:0:64825]      ucp_mm.c:1756 Assertion `mem_info.type == alloc_mem_type' failed: mem_info.mem_type=host alloc_mem_type=cuda
==== backtrace (tid:  64825) ====
 0  /__w/1/s/build-test/src/ucs/.libs/libucs.so.0(ucs_handle_error+0x12c) [0x7f216352e47c]
 1  /__w/1/s/build-test/src/ucs/.libs/libucs.so.0(ucs_fatal_error_message+0x50) [0x7f216352b4f0]

tvegas1 · 2025-03-19T18:31:54Z

==== Running ucx_info env vars test ====
+ UCX_IB_PORTS=mlx5_0:1
+ ./src/tools/info/ucx_info -epw -u t
+ grep -q -E UCX_IB_PORTS
+ grep unused
[swx-rdmz-ucx-gpu-02:64825:0:64825]      ucp_mm.c:1756 Assertion `mem_info.type == alloc_mem_type' failed: mem_info.mem_type=host alloc_mem_type=cuda
==== backtrace (tid:  64825) ====
 0  /__w/1/s/build-test/src/ucs/.libs/libucs.so.0(ucs_handle_error+0x12c) [0x7f216352e47c]
 1  /__w/1/s/build-test/src/ucs/.libs/libucs.so.0(ucs_fatal_error_message+0x50) [0x7f216352b4f0]

happening because of retaining a non-active apparently, will investigate.

tvegas1 · 2025-03-19T18:33:21Z

src/ucs/sys/topo/base/topo.c

+    return UCS_OK;
+}
+
+ucs_status_t ucs_topo_sys_dev_set_user_value(ucs_sys_device_t sys_dev,


brminich · 2025-03-19T19:35:46Z

src/ucs/sys/topo/base/topo.c

+    size_t length = ucs_static_array_size(ucs_topo_global_ctx.devices);
+    ucs_topo_sys_device_info_t *device;
+
+    ucs_carray_for_each(device, ucs_topo_global_ctx.devices, length) {
+        device->user_value = UINTPTR_MAX;
+    }


why needed? we do not pre-initialize other fields

brminich · 2025-03-19T19:36:13Z

src/ucs/sys/topo/base/topo.h

+ * Find system device by pci bus id.
+ *
+ * @param [in]  bus_id     pointer to bus id of the device of interest.
+ * @param [in]  user_value user_value to add along


Suggested change

* @param [in] user_value user_value to add along

* @param [in] user_value user_value to add along.

for consistency

brminich · 2025-03-19T19:42:35Z

src/uct/base/uct_mem.c

+    return (mem_type == UCS_MEMORY_TYPE_HOST) &&
+           (sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN);


do we really need it? I'd just ignore sys_dev if mem_type is host

seems it's not strictly needed. @yosefe, i assume this is for future proof?

yes, IMO better keep it for strict behavior

What is strict behavior? With this change we do not treat host memory as host anymore if sys_device is set. Then how should we handle such case? Imo, better to ignore sys_dev when memory type is host

if sys_device is set, we cannot do a host memory allocation because it would not match the system device requirement.

so we can return invalid param then. What will be the behavior with this change?

brminich · 2025-03-19T19:45:16Z

src/uct/cuda/base/cuda_md.c

@@ -74,7 +94,7 @@ uct_cuda_base_query_md_resources(uct_component_t *component,
    int i, num_gpus;

    status = UCT_CUDADRV_FUNC(cuDeviceGetCount(&num_gpus), UCS_LOG_LEVEL_DIAG);
-    if ((status != UCS_OK) || (num_gpus == 0)) {
+    if (status != UCS_OK) {


why need to remove check for num_gpus?

fixed thanks!

src/ucs/sys/topo/base/topo.c

yosefe · 2025-03-20T08:43:05Z

src/uct/base/uct_mem.c

+    return (mem_type == UCS_MEMORY_TYPE_HOST) &&
+           (sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN);


yes, IMO better keep it for strict behavior

src/uct/cuda/base/cuda_md.c

yosefe · 2025-03-20T08:45:44Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

-     "Do not use CUDA primary context if not active when allocating memory.\n",
-     ucs_offsetof(uct_cuda_copy_md_config_t, skip_inactive_primary_ctx),
+    {"RETAIN_INACTIVE_PRIMARY_CTX", "y",
+     "Also use inactive CUDA primary context for memory allocation\n",


Enable retaining and using an inactive CUDA primary context for memory allocation
remote trailing \n

src/uct/cuda/cuda_copy/cuda_copy_md.c

yosefe · 2025-03-20T08:49:55Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

    }

-    return UCS_OK;
+    if (*cu_device != CU_DEVICE_INVALID) {


add comment before:
/* Use the active cuda device */

added line before if(){}

src/uct/cuda/cuda_copy/cuda_copy_md.c

test/gtest/uct/cuda/test_switch_cuda_device.cc

yosefe · 2025-03-20T08:55:06Z

test/gtest/uct/cuda/test_switch_cuda_device.cc

+        for (auto device = 0; device < m_num_devices; ++device) {
+            ASSERT_EQ(cudaSetDevice(device), cudaSuccess);
+        }


can you pls add a comment to explain it?

tvegas1 · 2025-03-20T14:19:36Z

addressed comments and merged with common retain API, will push here when #10545 is in.

yosefe · 2025-03-21T08:26:32Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

@@ -79,7 +79,8 @@ static ucs_config_field_t uct_cuda_copy_md_config_table[] = {
     UCS_CONFIG_TYPE_ENUM(ucs_memory_type_names)},

    {"RETAIN_INACTIVE_PRIMARY_CTX", "y",
-     "Also use inactive CUDA primary context for memory allocation\n",
+     "Enable retaining and using an inactive CUDA primary context for memory"
+     " allocation\n",


remove trailing \n

yosefe

LGTM (to review merge fix)

tvegas1 · 2025-03-21T17:43:01Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

@@ -78,6 +78,12 @@ static ucs_config_field_t uct_cuda_copy_md_config_table[] = {
     ucs_offsetof(uct_cuda_copy_md_config_t, cuda_async_mem_type),
     UCS_CONFIG_TYPE_ENUM(ucs_memory_type_names)},

+    {"RETAIN_INACTIVE_PRIMARY_CTX", "n",


set to no to see CI pass for now

Maybe rename this variable to
RETAIN_INACTIVE_PRIMARY_CTX
retain_primary_ctx

to make code more compact
@rakhmets WDYT?

suggest renaming to retain_primary_ctx

rakhmets · 2025-03-21T18:05:52Z

src/ucs/sys/topo/base/topo.h

+ *
+ * @return UCS_OK or error in case device cannot be found.
+ */
+ucs_status_t ucs_topo_find_device_by_bus_id_user_value(


Maybe introduce void ucs_topo_sys_dev_set_user_value(ucs_sys_device_t sys_dev, uintptr_t user_value), and set it to UINTPTR_MAX in ucs_topo_find_device_by_bus_id.
Instead of having two similar methods ucs_topo_find_device_by_bus_id and ucs_topo_find_device_by_bus_id_user_value.

I started as you suggested and changed to address code reviews. @yosefe what do we do?

rakhmets · 2025-03-21T18:15:01Z

src/uct/cuda/base/cuda_md.c

+    }
+
+    *device = user_value;
+    if (*device == CU_DEVICE_INVALID) {


Looks like an unexpected case.
user_value is set only if all previous methods related to the device have been successfully completed.
Maybe assert is better for this case.

rakhmets · 2025-03-21T18:16:22Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+     "Enable retaining and using an inactive CUDA primary context for memory"
+     " allocation",


Enable seems to be redundant.
Retaining and use an inactive CUDA primary context for memory allocation

so: "Retain and use an inactive CUDA primary context for memory allocation"

rakhmets · 2025-03-21T18:18:40Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

@@ -405,7 +522,9 @@ uct_cuda_copy_mem_alloc(uct_md_h uct_md, size_t *length_p, void **address_p,
    *memh_p    = alloc_handle;
    *address_p = (void*)alloc_handle->ptr;
    *length_p  = alloc_handle->length;
-    return UCS_OK;
+out:
+    uct_cuda_copy_pop_ctx(cu_device, alloc_cu_device, UCS_LOG_LEVEL_ERROR);


WARN should be ok here. Since UCX can continue running.

rakhmets · 2025-03-21T18:20:50Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+        status = UCT_CUDADRV_FUNC(cuCtxPopCurrent(&ctx), log_level);
+        if (status != UCS_OK) {
+            return;
+        }
+
+        status = UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRelease(device), log_level);
+        if (status != UCS_OK) {
+            return;
+        }


Suggested change

status = UCT_CUDADRV_FUNC(cuCtxPopCurrent(&ctx), log_level);

if (status != UCS_OK) {

return;

}

status = UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRelease(device), log_level);

if (status != UCS_OK) {

return;

}

UCT_CUDADRV_FUNC(cuCtxPopCurrent(NULL), log_level);

UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRelease(device), log_level);

No need to exit if pop failed. We may warn user about the error, and it is better to try release the primary context on the device anyway.

rakhmets · 2025-03-21T18:21:49Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+    ucs_status_t status;
+    int dev, num_devices;
+
+    status = UCT_CUDADRV_FUNC(cuCtxGetDevice(cu_device_p), UCS_LOG_LEVEL_DEBUG);


Suggested change

status = UCT_CUDADRV_FUNC(cuCtxGetDevice(cu_device_p), UCS_LOG_LEVEL_DEBUG);

status = UCT_CUDADRV_FUNC_LOG_DEBUG(cuCtxGetDevice(cu_device_p));

rakhmets · 2025-03-21T18:32:50Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+    status = UCT_CUDADRV_FUNC(cuDeviceGetCount(&num_devices),
+                              UCS_LOG_LEVEL_DIAG);
+    if (status != UCS_OK) {
+        return UCS_ERR_INVALID_PARAM;
+    }
+
+    /* Use the first active cuda device for allocation */
+    for (dev = 0; dev < num_devices; dev++) {
+        if (UCT_CUDADRV_FUNC_LOG_DEBUG(cuDeviceGet(alloc_cu_device_p, dev)) !=
+            UCS_OK) {
+            continue;
+        }
+
+        status = uct_cuda_copy_push_ctx(md, *alloc_cu_device_p, log_level);
+        if (status == UCS_OK) {
+            break;
+        }
+    }
+
+    if (status != UCS_OK) {
+        ucs_log(log_level,
+                "no active cuda primary context for memory allocation");
+    }
+
+    return status;


Does this combination of methods from #10564 work for you?

static ucs_status_t uct_cuda_primary_ctx_push_first_active(CUdevice *cuda_device_p, CUcontext *cuda_ctx_p) { int num_devices, device_num; ucs_status_t status; CUdevice device; CUcontext ctx; status = UCT_CUDADRV_FUNC_LOG_ERR(cuDeviceGetCount(&num_devices)); if (status != UCS_OK) { return status; } for (device_num = 0; device_num < num_devices; ++device_num) { status = UCT_CUDADRV_FUNC_LOG_ERR(cuDeviceGet(&device, device_num)); if (status != UCS_OK) { return status; } status = uct_cuda_primary_ctx_retain(device, 0, &ctx); if (status == UCS_OK) { break; } else if (status == UCS_ERR_NO_DEVICE) { continue; } else { return status; } } if (device_num == num_devices) { return UCS_ERR_NO_DEVICE; } status = UCT_CUDADRV_FUNC_LOG_ERR(cuCtxPushCurrent(ctx)); if (status != UCS_OK) { UCT_CUDADRV_FUNC_LOG_WARN(cuDevicePrimaryCtxRelease(device)); return status; } *cuda_device_p = device; *cuda_ctx_p = ctx; return UCS_OK; } static UCS_F_ALWAYS_INLINE void uct_cuda_primary_ctx_pop_and_release(CUdevice cuda_device) { if (ucs_likely(cuda_device == CU_DEVICE_INVALID)) { return; } UCT_CUDADRV_FUNC_LOG_WARN(cuCtxPopCurrent(NULL)); UCT_CUDADRV_FUNC_LOG_WARN(cuDevicePrimaryCtxRelease(cuda_device)); }

yes agree, maybe we can merge this one and #10564 could extract the block? I understand this one might block other PR?

Let's keep both independent and after merge we unite the code

yosefe · 2025-03-22T09:41:09Z

src/uct/base/uct_mem.c

+    return (mem_type == UCS_MEMORY_TYPE_HOST) &&
+           (sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN);


if sys_device is set, we cannot do a host memory allocation because it would not match the system device requirement.

yosefe · 2025-03-22T09:42:59Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+     "Enable retaining and using an inactive CUDA primary context for memory"
+     " allocation",


so: "Retain and use an inactive CUDA primary context for memory allocation"

src/uct/cuda/cuda_copy/cuda_copy_md.c

test/gtest/uct/cuda/test_switch_cuda_device.cc

yosefe · 2025-03-22T09:59:12Z

test/gtest/uct/cuda/test_switch_cuda_device.cc

+UCS_TEST_P(test_mem_alloc_device, different_device_cuda_fabric,
+           "CUDA_COPY_ENABLE_FABRIC=y")


maybe add CUDA_COPY_ENABLE_FABRIC=y as a test variant?

i did not find existing infra in uct

see

ucx/test/gtest/uct/ib/test_dc.cc

Line 519 in eb1c7f3

enum_resources(const std::string &tl_name)

test/gtest/uct/test_mem.cc

yosefe · 2025-03-22T14:12:17Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+        (void)UCT_CUDADRV_FUNC(cuCtxPopCurrent(&ctx), UCS_LOG_LEVEL_WARN);
+        (void)UCT_CUDADRV_FUNC(cuDevicePrimaryCtxRelease(alloc_cu_device),
+                               UCS_LOG_LEVEL_WARN);


IMO it was better when these 2 calls were in a helper function uct_cuda_copy_pop_ctx because it is the opposite of uct_cuda_copy_push_ctx
then reuse uct_cuda_copy_pop_ctx also in uct_cuda_copy_md_sync_memops_get_address_range for consistency

yosefe · 2025-03-22T14:16:26Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+        if (uct_cuda_copy_push_ctx(cuda_device, 0, UCS_LOG_LEVEL_ERROR) !=
            UCS_OK) {
            return;
        }
-    } else {
-        cuda_mem_ctx = cuda_ctx;
-    }
-
-    if (UCT_CUDADRV_FUNC_LOG_ERR(cuCtxPushCurrent(cuda_mem_ctx)) != UCS_OK) {
-        goto out_primary_ctx_release;
+    } else if (UCT_CUDADRV_FUNC_LOG_ERR(cuCtxPushCurrent(cuda_ctx)) !=
+               UCS_OK) {
+        return;


if (cuda_ctx == NULL) { status = uct_cuda_copy_push_ctx(cuda_device, 0, UCS_LOG_LEVEL_ERROR); } else { status = UCT_CUDADRV_FUNC_LOG_ERR(cuCtxPushCurrent(cuda_ctx)); } if (status !=UCS_OK) { return; }

yosefe · 2025-03-22T14:19:57Z

test/gtest/uct/cuda/test_switch_cuda_device.cc

+UCS_TEST_P(test_mem_alloc_device, different_device_cuda_fabric,
+           "CUDA_COPY_ENABLE_FABRIC=y")


see

ucx/test/gtest/uct/ib/test_dc.cc

Line 519 in eb1c7f3

enum_resources(const std::string &tl_name)

yosefe · 2025-03-22T14:20:38Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

@@ -78,6 +78,12 @@ static ucs_config_field_t uct_cuda_copy_md_config_table[] = {
     ucs_offsetof(uct_cuda_copy_md_config_t, cuda_async_mem_type),
     UCS_CONFIG_TYPE_ENUM(ucs_memory_type_names)},

+    {"RETAIN_INACTIVE_PRIMARY_CTX", "n",


suggest renaming to retain_primary_ctx

brminich · 2025-03-23T16:32:05Z

src/uct/cuda/cuda_copy/cuda_copy_md.c

+{
+    CUcontext ctx;
+
+    (void)UCT_CUDADRV_FUNC(cuCtxPopCurrent(&ctx), UCS_LOG_LEVEL_WARN);


minor, can pass NULL instead of ctx

Co-authored-by: Thomas Vegas <[email protected]>

rakhmets reviewed Mar 14, 2025

View reviewed changes

src/uct/cuda/cuda_copy/cuda_copy_md.c Outdated Show resolved Hide resolved

brminich reviewed Mar 14, 2025

View reviewed changes

brminich reviewed Mar 17, 2025

View reviewed changes

rakhmets reviewed Mar 18, 2025

View reviewed changes

src/uct/cuda/cuda_copy/cuda_copy_md.c Outdated Show resolved Hide resolved

yosefe reviewed Mar 19, 2025

View reviewed changes

tvegas1 commented Mar 19, 2025

View reviewed changes

brminich reviewed Mar 19, 2025

View reviewed changes

yosefe reviewed Mar 20, 2025

View reviewed changes

yosefe reviewed Mar 21, 2025

View reviewed changes

tvegas1 commented Mar 21, 2025

View reviewed changes

rakhmets reviewed Mar 21, 2025

View reviewed changes

yosefe reviewed Mar 22, 2025

View reviewed changes

brminich previously approved these changes Mar 23, 2025

View reviewed changes

yosefe dismissed brminich’s stale review via 0d124c5 March 23, 2025 21:00

brminich approved these changes Mar 24, 2025

View reviewed changes

UCT/API/CUDA: Support allocating memory on a specific system device

200aa7e

Co-authored-by: Thomas Vegas <[email protected]>

yosefe force-pushed the uct_sys_dev branch from 0d124c5 to 200aa7e Compare March 24, 2025 07:36

yosefe enabled auto-merge March 24, 2025 07:37

		if ((mem_type != UCS_MEMORY_TYPE_HOST) \|\|
		(sys_dev != UCS_SYS_DEVICE_ID_UNKNOWN)) {

	* @param [in] user_value user_value to add along
	* @param [in] user_value user_value to add along.

		return (mem_type == UCS_MEMORY_TYPE_HOST) &&
		(sys_dev == UCS_SYS_DEVICE_ID_UNKNOWN);

		"Enable retaining and using an inactive CUDA primary context for memory"
		" allocation",

	status = UCT_CUDADRV_FUNC(cuCtxGetDevice(cu_device_p), UCS_LOG_LEVEL_DEBUG);
	status = UCT_CUDADRV_FUNC_LOG_DEBUG(cuCtxGetDevice(cu_device_p));

		UCS_TEST_P(test_mem_alloc_device, different_device_cuda_fabric,
		"CUDA_COPY_ENABLE_FABRIC=y")

UCT/CUDA: Use sys_dev to allocate CUDA memory #10553

Are you sure you want to change the base?

UCT/CUDA: Use sys_dev to allocate CUDA memory #10553

Conversation

tvegas1 commented Mar 14, 2025

What?

Why?

How?

tvegas1 commented Mar 14, 2025

rakhmets commented Mar 14, 2025

rakhmets Mar 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvegas1 commented Mar 17, 2025

Choose a reason for hiding this comment

tvegas1 commented Mar 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosefe commented Mar 19, 2025

tvegas1 commented Mar 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tvegas1 commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosefe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhmets Mar 14, 2025 •

edited

Loading