[Linux] Process.open_files() crashes if an open file is not accessible

[michaellass]

Summary

• OS: Linux
• Architecture: 64bit
• Psutil version: 5.9.1
• Python version: Python 3.10.5
• Type: core

Description

Process.open_files() crashes if an open file is not accessible

Take the following reproducer:

#!/usr/bin/env python3

import os
import psutil

print(psutil.version_info)

dir = "testdir"
if not os.path.exists(dir):
    os.mkdir(dir)

with open(os.path.join(dir, "somefile"), "w+") as file:
    os.chmod(dir, 644)
    try:
        print(psutil.Process().open_files())
    finally:
        os.chmod(dir, 755)

Expected result: Gracefully handle inaccessible files, as callers of open_files do not expect any exceptions.

Actual result: An exception is thrown. While handling the exception, another exception is thrown.

Output of the reproducer:

(5, 9, 1)
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/psutil/_pslinux.py", line 1642, in wrapper
    return fun(self, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/psutil/_pslinux.py", line 2209, in open_files
    if path.startswith('/') and isfile_strict(path):
  File "/usr/lib/python3.10/site-packages/psutil/_common.py", line 481, in isfile_strict
    st = os.stat(path)
PermissionError: [Errno 13] Permission denied: '/home/bevan/psutil-test/testdir/somefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bevan/psutil-test/./repro.py", line 15, in <module>
    print(psutil.Process().open_files())
  File "/usr/lib/python3.10/site-packages/psutil/__init__.py", line 1142, in open_files
    return self._proc.open_files()
  File "/usr/lib/python3.10/site-packages/psutil/_pslinux.py", line 1644, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: (pid=99424)

The condition under which this error occurs may seem quite odd. Why should a process have an open file descriptor to a file that it cannot see? Turns out that this is a practical issue when your python process is launched from a process that entered a Linux namespace via setns(2), the bind mount referencing that namespace is only accessible to root and privileges are dropped before launching python. This happens in systems that manage and isolate user processes from each other, such as the SLURM workload manager used in many HPC clusters.

[giampaolo]

The code checks whether the path is a regular file:

psutil/psutil/_pslinux.py

Lines 2205 to 2209 in c13017f

	# If path is not an absolute there's no way to tell
	# whether it's a regular file or not, so we skip it.
	# A regular file is always supposed to be have an
	# absolute path though.
	if path.startswith('/') and isfile_strict(path):

The check is needed because /proc/<PID>/fd can return all sorts of file descriptor types (pipes , unix sockets, directories, ...), but open_files() is designed to return regular files only, so unfortunately there is no way to avoid EPERM.

After you run your script also ls returns EPERM:

$ ls -l testdir/
ls: cannot open directory 'testdir/': Permission denied

...meaning this is a standard UNIX permission issue and shouldn't be considered a psutil bug.

[michaellass]

Of course ls -l testdir returns EPERM as the directory is not accessible but is a python program using Process.open_files() to get a list of open file descriptions really expected to crash with an exception here? An alternative would be to just skip inaccessible files similar to non-absolute paths, as their type cannot be determined.

[giampaolo]

Of course ls -l testdir returns EPERM as the directory is not accessible but is a python program using Process.open_files() to get a list of open file descriptions really expected to crash with an exception here?

Yes. The same way it's not accessible to ls it's also not accessible to psutil / python, because internally they use the same UNIX syscall (stat(2)) to determine the file type. Both programs interrupt execution due to insufficient permissions. They do so because there really isn't anything they can do about it except signal what went wrong and give up. Getting an exception doesn't necessarily mean something is broken per se.

An alternative would be to just skip inaccessible files similar to relative paths, as their type cannot be determined.

As a general rule, psutil tends to avoid returning incomplete results and prefers to "crash" with AccessDenied instead. This is good because this way you can rely on open_files() (and other APIs), and assume it always tells the truth. If it didn't, you'd have a bug in your code and you probably wouldn't know it. This is why the Python Zen says that "errors should never pass silently" basically. =)

With that said, it is true that sometimes it is desirable to get incomplete results, as it seems this is the case. Perhaps open_files() can grow an extra parameter to allow for this. Something like:

def open_files(ignore_eperm=False)

...and perhaps psutil.net_connections() and psutil.Process.connections() can do the same. This should be implemented across all platforms though, not only on Linux. I'm going to re-open this ticket as it seems the use case is legitimate.

[michaellass]

Thanks a lot for the detailed reasoning. Knowing that the current behavior is actually expected, I'll try to get those users who stumbled across this behavior involved.

An API extension such as an ignore_eperm argument might be a viable solution. Compared to using try/except in the calling code, this would allow users to get a known-incomplete list of currently opened files.

Thanks for reopening. I'll try to get people directly affected by this issue involved in this discussion.

[giampaolo]

Proposal: #2329.