pytest hangs if it tries to rewrite modules without having access rights on Windows 10

[Sup3rGeo]

Basically, it seems that if:

• python is installed in C:\Program files (in a situation that you also have to run pip install from an elevated command prompt in order to install packages)
• pytest runs (as a normal user) for the first time and tries to rewrite assertions, without having elevation

Then pytest just hangs. CTRL+C shows the following traceback:

Traceback (most recent call last):
  File "c:\program files\python36\lib\tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python36\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 55, in main
    config = _prepareconfig(args, plugins)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 200, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "c:\program files\python36\lib\site-packages\pluggy\hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 87, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 81, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 203, in _multicall
    gen.send(outcome)
  File "c:\program files\python36\lib\site-packages\_pytest\helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "c:\program files\python36\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 661, in pytest_cmdline_parse
    self.parse(args)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 869, in parse
    self._preparse(args, addopts=addopts)
  File "c:\program files\python36\lib\site-packages\_pytest\config\__init__.py", line 815, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "c:\program files\python36\lib\site-packages\pluggy\manager.py", line 292, in load_setuptools_entrypoints
    plugin = ep.load()
  File "c:\program files\python36\lib\site-packages\importlib_metadata\__init__.py", line 90, in load
    module = import_module(match.group('module'))
  File "c:\program files\python36\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 144, in exec_module
    _write_pyc(state, co, source_stat, pyc)
  File "c:\program files\python36\lib\site-packages\_pytest\assertion\rewrite.py", line 267, in _write_pyc
    with atomicwrites.atomic_write(pyc, mode="wb", overwrite=True) as fp:
  File "c:\program files\python36\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 156, in _open
    with get_fileobject(**self._open_kwargs) as f:
  File "c:\program files\python36\lib\site-packages\atomicwrites\__init__.py", line 172, in get_fileobject
    descriptor, name = tempfile.mkstemp(dir=dir)
  File "c:\program files\python36\lib\tempfile.py", line 344, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "c:\program files\python36\lib\tempfile.py", line 268, in _mkstemp_inner
    if (_os.name == 'nt' and _os.path.isdir(dir) and
KeyboardInterrupt

I would expect it not to hang, but for it to terminate with a Permission denied message instead.

Using Windows 10, python 3.6.8 and pytest 5.0.0

[nicoddemus]

Hi @Sup3rGeo!

atomicwrites tries to create the temporary directory 'c:\\program files\\python36\\lib\\site-packages\\typhoon\\__pycache__\\tmpu12vgv2e' using tempfile.mkstemp, which in turn calls _mkstemp_inner:

def _mkstemp_inner(dir, pre, suf, flags, output_type):
    """Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""

    names = _get_candidate_names()
    if output_type is bytes:
        names = map(_os.fsencode, names)

    for seq in range(TMP_MAX):
        name = next(names)
        file = _os.path.join(dir, pre + name + suf)
        try:
            fd = _os.open(file, flags, 0o600)
        except FileExistsError:
            continue    # try again
        except PermissionError:
            # This exception is thrown when a directory with the chosen name
            # already exists on windows.
            if (_os.name == 'nt' and _os.path.isdir(dir) and
                _os.access(dir, _os.W_OK)):
                continue
            else:
                raise
        return (fd, _os.path.abspath(file))

    raise FileExistsError(_errno.EEXIST,
                          "No usable temporary file name found")

The traceback shows it is hanging on the PermissionError handling, which does call _os.access(dir, _os.W_OK) to check if it should keep retrying to create the temporary directory, otherwise it fails. It seems for some reason it is keeping looping there, instead of _os.access(dir, _os.W_OK) returning False and we getting the expected PermissionError.

Strange, unfortunately this is hard to debug. 😕

[sparrowt]

Hopefully #4730 would help (see also #4755)
Edit: now done in #5864

[Sup3rGeo]

According to this stackoverflow question, it seems this is a long lasting issue for windows that has not been yet fixed:

https://bugs.python.org/issue22107

Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.

[nicoddemus]

Let's hope the items mentioned by @sparrowt above will help workaround the issue for pytest.

It would definitely help, but only for py38 users I'm afraid... 🤔

[Blubbaa]

sadly this issue still exists on Windows 10, Python 3.8.6, pytest 6.1.2 😖

[nicoddemus]

I guess a workaround would be to check for permission before calling into atomic_write.

I've spent a few minutes trying to write a test for it:

@pytest.mark.skipif(not sys.platform.startswith("win"), reason="Windows only (#5844)")
def test_write_pyc(tmp_path: Path, request) -> None:
    cache_dir = tmp_path.joinpath("cache")
    cache_dir.mkdir()

    try:
        cache_dir.chmod(0)  # this doesn't work
        with pytest.raises(PermissionError):
            cache_dir.joinpath("foo").touch()
        # call _write_pyc and check it returns False instead of hanging
    finally:
        cache_dir.chmod(stat.S_IWRITE)

But unfortunately couldn't get cached_dir to throw a PermissionError for some reason. I'm leaving this here in case someone has some tips, as I'm short on time right now to investigate further. 👍

[marcel-kanter]

Stumbled across this too... There are two issues:

1. The assumption with the permission error
2. the pyc-path may be read-only for normal users (this may be on linux too?)
3. get_fileobject of atomicwrites gets called without a dir from assertion rewrite

os.fspath(pyc) may be read-only (on linux too?)
and this arguments does not go to get_fileobject (this is called with dir empty) and then is replaces it (which is dangerous, since this may be read-only too)

    if dir is None:
        dir = os.path.normpath(os.path.dirname(self._path))

printing dir after these two lines in atomicwrites/init.py

gives me C:\Program Files\Python38\lib\site-packages\hypothesis_pycache_

with is write protected.

File "C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py", line 165, in exec_module
_write_pyc(state, co, source_stat, pyc)
File "C:\Program Files\Python38\lib\site-packages_pytest\assertion\rewrite.py", line 308, in write_pyc
with atomic_write(os.fspath(pyc), mode="wb", overwrite=True) as fp:
File "C:\Program Files\Python38\lib\contextlib.py", line 113, in enter
return next(self.gen)
File "C:\Program Files\Python38\lib\site-packages\atomicwrites_init.py", line 166, in _open
with get_fileobject(**self.open_kwargs) as f:
File "C:\Program Files\Python38\lib\site-packages\atomicwrites_init.py", line 190, in get_fileobject

Calling pytest with --assert=plain is a work-around.

[crazyjurich]

Still broken on Win10, Python 3.9.6, pytest-6.2.4
Calling pytest with --assert=plain indeed works.

And in my case pytest was trying to rewrite allure-pytest module under site-packages. I don't think pytest has to rewrite libraries.
On my other env with python 3.8 pytest is working Ok, I have found that python itself is owned by system and some folders under site-packages are owned by my user, this is the reason of not failing.

Did some more experiments and found out that:

• If I install modules with pip under elevated power shell (python is under program files), folder under site-packages is owned by system (which is correct) and pytest fails to write his PYCs
• If I install modules with pip under elevated git bash (python is under program files), folder under site-packages is owned by user (which is incorrect) and pytest works. So this is one of workarounds

And I think other workaround is not installing python under program files, but under your user. Didn't test it though.

IMHO pytest should detect if it has write access, and either skip rewriting for modules he doesn't have access to, either write PYCs somewhere else in this case, either don't rewrite libraries within python at all.

[marcel-kanter]

Another workaround maybe to use virtual environments (which have the user's rights).

But how do the Linux guys handle this? They should get the same problem...

[notgriffin]

Ran into this issue but couldn't use virtual environments out of the box since I am using a program that depends on using system-wide installed packages. The solution ended up being installing all my needed packages to the system and then creating a virtual environment with the command virtualenv --system-site-packages to use all of the system packages. Then I installed pytest on its own to the virtual environment with pip install --ignore-installed pytest. This forced pytest to not be able to write to system-installed packages. Hopefully this workaround will help someone out. Credit: https://stackoverflow.com/a/19459977

[pawelszramowski]

I have also been experiencing this issue (currently on Windows 10, Python 3.9.7, and pytest 7.1.1).

I've observed that it happens after I've used pip to update packages. My workaround is to delete all the __pycache__ directories in the Python installation directory. Then pytest starts working even without elevated permissions.