[Ubuntu 20.04] cannot import psutil, circular import suspected

[StrangeTcy]

Summary

• OS: { Ubuntu 20.04 }
• Architecture: { 64bit }
• Psutil version: { unknown }
• Python version: { 3.8.10 }
• Type: { using another library }

Description

{{{
calling a python script with deepspeed eventually comes down to trying and failing to import psutil:

Traceback (most recent call last):
 File "/usr/local/bin/deepspeed", line 3, in <module>
   from deepspeed.launcher.runner import main
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/__init__.py", line 15, in <module>
   from . import module_inject
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/module_inject/__init__.py", line 3, in <module>
   from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/module_inject/replace_module.py", line 811, in <module>
   from ..pipe import PipelineModule
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/pipe/__init__.py", line 3, in <module>
   from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/runtime/pipe/__init__.py", line 3, in <module>
   from .module import PipelineModule, LayerSpec, TiedLayerSpec
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/runtime/pipe/module.py", line 15, in <module>
   from .. import utils as ds_utils
 File "/usr/local/lib/python3.11/dist-packages/deepspeed/runtime/utils.py", line 12, in <module>
   import psutil
 File "/usr/lib/python3/dist-packages/psutil/__init__.py", line 95, in <module>
   from . import _pslinux as _psplatform
 File "/usr/lib/python3/dist-packages/psutil/_pslinux.py", line 26, in <module>
   from . import _psutil_linux as cext
ImportError: cannot import name '_psutil_linux' from partially initialized module 'psutil' (most likely due to a circular import) (/usr/lib/python3/dist-packages/psutil/__init__.py)

}}}

[giampaolo]

You most likely have a conflict with a previous psutil installation. If you download psutil source you can run make clean and make uninstall, then reinstall. Usually that fixes this sort of error.

[StrangeTcy]

Nope.
git clone https://github.com/giampaolo/psutil.git; cd psutil; sudo make clean; sudo make uninstall; sudo python setup.py install; python -c "import psutil" -> same problem
sudo pip uninstall psutil; sudo python install -U psutil -> "it's a distutils project"
sudo pip uninstall psutil; cd psutil; sudo python setup.py install; python -c "import psutil" -> same problem

[Timingan]

I encountered the same issue. May I ask how you fixed this?

[danepowell]

I'm hitting the same problem trying to import psutil for use on my own project. My project is running on the Steam Deck using python and the version of psutil as shipped with the system, so I don't think it's possible that I've broken it in any way.