Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shm transport causes tests to hang #780

Open
pentschev opened this issue Sep 10, 2021 · 1 comment
Open

shm transport causes tests to hang #780

pentschev opened this issue Sep 10, 2021 · 1 comment

Comments

@pentschev
Copy link
Member

Some of the larger tests below will hang indefinitely when shm is enabled, but removing that transport and leaving only tcp succeeds, enabling other transports such as rc and posix is also fine.

UCX_TLS=shm,tcp,cuda_copy pytest -vs --cache-clear tests/test_send_recv.py

It's not clear whether this is a bug in UCX, but it's possible that support for shm in blocking mode isn't stable. We need to test #565 again.

@pentschev
Copy link
Member Author

This seems to be working with UCX master:

$ ucx_info -v
# UCT version=1.12.0 revision af82548
# configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --prefix=/datasets/pentschev/miniconda3/envs/rn-112-21.12.211025 --with-cuda=/usr/local/cuda-11.2 --enable-mt
$ UCX_TLS=tcp,cuda_copy,shm pytest tests/test_send_recv.py
================================================= test session starts ==================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /datasets/pentschev/simple-ci/src/ucx-py
plugins: asyncio-0.16.0, repeat-0.8.0, rerunfailures-10.2
collected 146 items

tests/test_send_recv.py ........................................................................................ [ 60%]
..........................................................                                                       [100%]

================================================= 146 passed in 45.12s =================================================

cc @madsbk

@caryr35 caryr35 added this to ucx-py Jan 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant