Covalent commands stuck without any way to debug

[pradeepmantha]

Environment

• Covalent version:covalent-0.220.0.post2
• Python version: 3.8
• Operating system: SUSE Enterprise 15

What is happening?

Any covalent command stucks
python
Python 3.8.18 (default, Sep 11 2023, 13:40:15)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import covalent
covalent.version
^C^CTraceback (most recent call last):
File "", line 1, in
File "/py38/lib/python3.8/site-packages/covalent/init.py", line 22, in
from . import executor, leptons # nopycln: import
File "py38/lib/python3.8/site-packages/covalent/executor/init.py", line 32, in
from .._shared_files import logger
File "/py38/lib/python3.8/site-packages/covalent/_shared_files/logger.py", line 30, in
log_level = get_config("sdk.log_level").upper()
File "/py38/lib/python3.8/site-packages/covalent/_shared_files/config.py", line 243, in get_config
cm = ConfigManager()
File "/py38/lib/python3.8/site-packages/covalent/_shared_files/config.py", line 51, in init
self.update_config()
File "py38/lib/python3.8/site-packages/covalent/_shared_files/config.py", line 113, in update_config
fcntl.lockf(f, fcntl.LOCK_EX)

How can we reproduce the issue?

Test on SUSE enterprise with latest version of covalent and py3.8

What should happen?

covalent start will just stuck for long time

Any suggestions?

No response

[Andrew-S-Rosen]

This suggests that the COVALENT_CONFIG_DIR is set to a location that does not support file locking. See #1697.

[santoshkumarradha]

This suggests that the COVALENT_CONFIG_DIR is set to a location that does not support file locking. See #1697.

Yes this does seem like config file smell. @kessler-frost can you add this in?

[Andrew-S-Rosen]

I will note that @pradeepmantha direct messaged me the following:

I was able to bring up covalent server on the perlmutter compute node successfully after changing the config directory.