ImportWarning: can't resolve package from spec or package #7824

ghost · 2020-09-30T17:02:59Z

ghost
Sep 30, 2020

Hi

Looks like the issue already reported in #3061 and #4423 is back with pytest version 6.1.0 (issue instantly disappears when downgrading to pytest==6.0.2).

Details:

Setup:

# python --version
Python 3.6.12
# pip freeze | grep pytest
pytest==6.1.0
pytest-cov==2.10.1
pytest-mock==3.3.1
pytest-sugar==0.9.4

Encountered traceback:

# pytest
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 187, in console_main
    code = main()
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 319, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.6/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config = outcome.get_result()  # type: Config
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1280, in parse
    self._preparse(args, addopts=addopts)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1187, in _preparse
    early_config=self, args=args, parser=self._parser
  File "/usr/local/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.6/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/local/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 1070, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(early_config.known_args_namespace)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 503, in _set_initial_conftests
    self._try_load_conftest(current, namespace.importmode)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 513, in _try_load_conftest
    self._getconftestmodules(x, importmode)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 536, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 587, in _importconftest
    self.consider_conftest(mod)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 663, in consider_conftest
    self.register(conftestmodule, name=conftestmodule.__file__)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 443, in register
    self.consider_module(plugin)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 669, in consider_module
    self._import_plugin_specs(getattr(mod, "pytest_plugins", []))
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 676, in _import_plugin_specs
    self.import_plugin(import_spec)
  File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 703, in import_plugin
    __import__(importspec)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "/usr/local/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 171, in exec_module
    exec(co, module.__dict__)
  File "/lib/data-science/tests/fixtures.py", line 12, in <module>
    from sklearn.ensemble import RandomForestClassifier
  File "/usr/local/lib/python3.6/site-packages/sklearn/__init__.py", line 134, in <module>
    from .base import clone
  File "/usr/local/lib/python3.6/site-packages/sklearn/base.py", line 13, in <module>
    from .utils.fixes import signature
  File "/usr/local/lib/python3.6/site-packages/sklearn/utils/__init__.py", line 10, in <module>
    from .murmurhash import murmurhash3_32
  File "sklearn/utils/murmurhash.pyx", line 19, in init sklearn.utils.murmurhash
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path_

Answered by asottile

Oct 1, 2020

this was a bug in cython which has been fixed: cython/cython#1720

I'd suggest upgrading scikit-learn

View full answer

asottile · 2020-10-01T18:36:42Z

asottile
Oct 1, 2020
Collaborator

this was a bug in cython which has been fixed: cython/cython#1720

I'd suggest upgrading scikit-learn

0 replies

ghost · 2020-10-01T19:15:34Z

ghost
Oct 1, 2020

Maybe I should consider upgrading the version of scikit-learn used in the project, but this nevertheless really looks like a regression from pytest since simply downgrading it to version ==6.0.2 (without any other change) avoids the uncaught exception.
Assuming that the only cause of this issue is cython/cython#1720 (which is more than 3 years old), how do you explain that everything works fine with pytest==6.0.2?

0 replies

asottile · 2020-10-01T19:49:38Z

asottile
Oct 1, 2020
Collaborator

You can reproduce this without pytest:

python -W error -c 'import sklearn.utils'

it's likely that a warning was hidden which is now shown -- the warning is correct though

the reason this can happen is due to prebuilt wheels, wheels are never rebuilt on pypi after release

for example, I can install scikit-learn==0.20.4 (the last version I can see this reproducing on) and show the same problem:

$ pip install scikit-learn==0.20.4
Collecting scikit-learn==0.20.4
  Downloading scikit_learn-0.20.4-cp36-cp36m-manylinux1_x86_64.whl (5.4 MB)
     |████████████████████████████████| 5.4 MB 2.7 MB/s 
Requirement already satisfied: numpy>=1.8.2 in ./venv/lib/python3.6/site-packages (from scikit-learn==0.20.4) (1.19.2)
Requirement already satisfied: scipy>=0.13.3 in ./venv/lib/python3.6/site-packages (from scikit-learn==0.20.4) (1.5.2)
Installing collected packages: scikit-learn
  Attempting uninstall: scikit-learn
    Found existing installation: scikit-learn 0.21.0
    Uninstalling scikit-learn-0.21.0:
      Successfully uninstalled scikit-learn-0.21.0
Successfully installed scikit-learn-0.20.4
WARNING: You are using pip version 20.2.1; however, version 20.2.3 is available.
You should consider upgrading via the '/tmp/venv/bin/python -m pip install --upgrade pip' command.
$ python -Werror -c 'import sklearn.utils'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/venv/lib/python3.6/site-packages/sklearn/__init__.py", line 64, in <module>
    from .base import clone
  File "/tmp/venv/lib/python3.6/site-packages/sklearn/base.py", line 14, in <module>
    from .utils.fixes import signature
  File "/tmp/venv/lib/python3.6/site-packages/sklearn/utils/__init__.py", line 12, in <module>
    from .murmurhash import murmurhash3_32
  File "sklearn/utils/murmurhash.pyx", line 19, in init sklearn.utils.murmurhash
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

without pytest

0 replies

asottile · 2020-10-01T19:55:50Z

asottile
Oct 1, 2020
Collaborator

also the only way this could be triggered by default is you've upgraded warnings to errors (likely with pytest.ini or some config since I don't see any commandline arguments in your invocation)

0 replies

ghost · 2020-10-01T22:23:47Z

ghost
Oct 1, 2020

The change on warnings handling was the missing piece (I missed it in the change log); now everything make sense. Thanks for the explanation, and sorry for the noise :/

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImportWarning: can't resolve package from spec or package #7824

{{title}}

Replies: 5 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ImportWarning: can't resolve package from __spec__ or __package__ #7824

ghost Sep 30, 2020

Replies: 5 comments

asottile Oct 1, 2020 Collaborator

ghost Oct 1, 2020

asottile Oct 1, 2020 Collaborator

asottile Oct 1, 2020 Collaborator

ghost Oct 1, 2020

ImportWarning: can't resolve package from spec or package #7824

ghost
Sep 30, 2020

asottile
Oct 1, 2020
Collaborator

ghost
Oct 1, 2020

asottile
Oct 1, 2020
Collaborator

asottile
Oct 1, 2020
Collaborator

ghost
Oct 1, 2020