Merge branch 'master' into mx-psi/permissions-issue

.github/workflows/build.yml

@@ -16,7 +16,7 @@
 # * https://github.com/marketplace/actions/cancel-workflow-action
 # * https://github.com/vmactions/freebsd-vm
 
-on: [push]
+on: [push, pull_request]
 name: build
 jobs:
   linux-macos-win:

CREDITS

@@ -745,6 +745,10 @@ N: David Knaack
 W: https://github.com/davidkna
 I: 1921
 
+N: Nikita Radchenko
+W: https://github.com/nradchenko
+I: 1940
+
 N: MaWe2019
 W: https://github.com/MaWe2019
 I: 1953
@@ -758,3 +762,15 @@ D: fix typos in documentation
 N: Pablo Baeyens
 W: https://github.com/mx-psi
 I: 1598, 1974
+
+N: Xuehai Pan
+W: https://github.com/XuehaiPan
+I: 1948
+
+N: Saeed Rasooli
+W: https://github.com/ilius
+I: 1996
+
+N: PetrPospisil
+W: https://github.com/PetrPospisil
+I: 1980

HISTORY.rst

@@ -10,23 +10,48 @@ XXXX-XX-XX
 - 1851_: [Linux] cpu_freq() is slow on systems with many CPUs. Read current
   frequency values for all CPUs from /proc/cpuinfo instead of opening many
   files in /sys fs.  (patch by marxin)
+- 1992_: NoSuchProcess message now specifies if the PID has been reused.
+- 1992_: error classes (NoSuchProcess, AccessDenied, etc.) now have a better
+  formatted and separated `__repr__` and `__str__` implementations.
+- 1996_: add support for MidnightBSD.  (patch by Saeed Rasooli)
+- 1999_: [Linux] disk_partitions(): convert "/dev/root" device (an alias used
+  on some Linux distros) to real root device path.
 
 **Bug fixes**
 
 - 1456_: [macOS] psutil.cpu_freq()'s min and max are set to 0 if can't be
   determined (instead of crashing).
 - 1512_: [macOS] sometimes Process.connections() will crash with EOPNOTSUPP
   for one connection; this is now ignored.
+- 1598_: [Windows] psutil.disk_partitions() only returns mountpoints on drives
+  where it first finds one
+- 1874_: [Solaris] swap output error due to incorrect range.
 - 1892_: [macOS] psutil.cpu_freq() broken on Apple M1.
+- 1901_: [macOS] different functions, especially process' open_files() and
+  connections() methods, could randomly raise AccessDenied because the internal
+  buffer of `proc_pidinfo(PROC_PIDLISTFDS)` syscall was not big enough. We now
+  dynamically increase the buffer size until it's big enough instead of giving
+  up and raising AccessDenied, which was a fallback to avoid crashing.
 - 1904_: [Windows] OpenProcess fails with ERROR_SUCCESS due to GetLastError()
   called after sprintf().  (patch by alxchk)
-- 1874_: [Solaris] swap output error due to incorrect range.
 - 1913_: [Linux] wait_procs seemingly ignoring timeout, TimeoutExpired thrown
+- 1919_: [Linux] sensors_battery() can raise TypeError on PureOS.
 - 1921_: [Windows] psutil.swap_memory() shows committed memory instead of swap
+- 1940_: [Linux] psutil does not handle ENAMETOOLONG when accessing process
+  file descriptors in procfs.  (patch by Nikita Radchenko)
+- 1948_: Process' memoize_when_activated decorator was not thread-safe.  (patch
+  by Xuehai Pan)
 - 1953_: [Windows] disk_partitions() crashes due to insufficient buffer len.
   (patch by MaWe2019)
 - 1598_: [Windows] psutil.disk_partitions() only returns mountpoints on drives
   where it first finds one
+- 1965_: [Windows] fix "Fatal Python error: deallocating None" when calling
+  psutil.users() multiple times.
+- 1980_: [Windows] 32bit / WOW64 processes fails to read process name longer
+  than 128 characters resulting in AccessDenied. This is now fixed.  (patch
+  by PetrPospisil)
+- 1991_: process_iter() can raise TypeError if invoked from multiple threads
+  (not thread-safe).
 - 1974_: [Windows] psutil.disk_io_counters() raises permission error.
 
 5.8.0

INSTALL.rst

@@ -93,7 +93,7 @@ If you don't have pip you can install with wget::
 
     wget https://bootstrap.pypa.io/get-pip.py -O - | python3
 
-...ow with curl::
+...or with curl::
 
     python3 < <(curl -s https://bootstrap.pypa.io/get-pip.py)
 

README.rst

@@ -139,7 +139,9 @@ Supporters
       <a href="https://github.com/dbwiddis"><img height="40" width="40" title="Daniel Widdis" src="https://avatars1.githubusercontent.com/u/9291703?s=88&amp;v=4" /></a>
       <a href="https://github.com/aristocratos"><img height="40" width="40" title="aristocratos" src="https://avatars3.githubusercontent.com/u/59659483?s=96&amp;v=4" /></a>
       <a href="https://github.com/cybersecgeek"><img height="40" width="40" title="cybersecgeek" src="https://avatars.githubusercontent.com/u/12847926?v=4" /></a>
+      <a href="https://github.com/scoutapm-sponsorships"><img height="40" width="40" title="scoutapm-sponsorships" src="https://avatars.githubusercontent.com/u/71095532?v=4" /></a>
       <a href="https://opencollective.com/chenyoo-hao"><img height="40" width="40" title="Chenyoo Hao" src="https://images.opencollective.com/chenyoo-hao/avatar/40.png" /></a>
+      <a href="https://opencollective.com/alexey-vazhnov"><img height="40" width="40" title="Alexey Vazhnov" src="https://images.opencollective.com/alexey-vazhnov/daed334/avatar/40.png" /></a>
     </div>
     <sup><a href="https://github.com/sponsors/giampaolo">add your avatar</a></sup>
 

docs/index.rst

@@ -83,7 +83,9 @@ Supporters
       <a href="https://github.com/dbwiddis"><img height="40" width="40" title="Daniel Widdis" src="https://avatars1.githubusercontent.com/u/9291703?s=88&amp;v=4" /></a>
       <a href="https://github.com/aristocratos"><img height="40" width="40" title="aristocratos" src="https://avatars3.githubusercontent.com/u/59659483?s=96&amp;v=4" /></a>
       <a href="https://github.com/cybersecgeek"><img height="40" width="40" title="cybersecgeek" src="https://avatars.githubusercontent.com/u/12847926?v=4" /></a>
+      <a href="https://github.com/scoutapm-sponsorships"><img height="40" width="40" title="scoutapm-sponsorships" src="https://avatars.githubusercontent.com/u/71095532?v=4" /></a>
       <a href="https://opencollective.com/chenyoo-hao"><img height="40" width="40" title="Chenyoo Hao" src="https://images.opencollective.com/chenyoo-hao/avatar/40.png" /></a>
+      <a href="https://opencollective.com/alexey-vazhnov"><img height="40" width="40" title="Alexey Vazhnov" src="https://images.opencollective.com/alexey-vazhnov/daed334/avatar/40.png" /></a>
     </div>
     <br />
     <sup><a href="https://github.com/sponsors/giampaolo">add your avatar</a></sup>
@@ -554,7 +556,7 @@ Network
   numbers will always be increasing or remain the same, but never decrease.
   ``net_io_counters.cache_clear()`` can be used to invalidate the *nowrap*
   cache.
-  On machines with no network iterfaces this function will return ``None`` or
+  On machines with no network interfaces this function will return ``None`` or
   ``{}`` if *pernic* is ``True``.
 
     >>> import psutil
@@ -983,31 +985,37 @@ Exceptions
 .. class:: NoSuchProcess(pid, name=None, msg=None)
 
   Raised by :class:`Process` class methods when no process with the given
-  *pid* is found in the current process list or when a process no longer
+  *pid* is found in the current process list, or when a process no longer
   exists. *name* is the name the process had before disappearing
   and gets set only if :meth:`Process.name()` was previously called.
 
 .. class:: ZombieProcess(pid, name=None, ppid=None, msg=None)
 
   This may be raised by :class:`Process` class methods when querying a zombie
-  process on UNIX (Windows doesn't have zombie processes). Depending on the
-  method called the OS may be able to succeed in retrieving the process
-  information or not.
-  Note: this is a subclass of :class:`NoSuchProcess` so if you're not
-  interested in retrieving zombies (e.g. when using :func:`process_iter()`)
-  you can ignore this exception and just catch :class:`NoSuchProcess`.
+  process on UNIX (Windows doesn't have zombie processes).
+  *name* and *ppid* attributes are available if :meth:`Process.name()` or
+  :meth:`Process.ppid()` methods were called before the process turned into a
+  zombie.
+
+  .. note::
+
+    this is a subclass of :class:`NoSuchProcess` so if you're not interested
+    in retrieving zombies (e.g. when using :func:`process_iter()`) you can
+    ignore this exception and just catch :class:`NoSuchProcess`.
 
   .. versionadded:: 3.0.0
 
 .. class:: AccessDenied(pid=None, name=None, msg=None)
 
   Raised by :class:`Process` class methods when permission to perform an
-  action is denied. "name" is the name of the process (may be ``None``).
+  action is denied due to insufficient privileges.
+  *name* attribute is available if :meth:`Process.name()` was previously called.
 
 .. class:: TimeoutExpired(seconds, pid=None, name=None, msg=None)
 
-  Raised by :meth:`Process.wait` if timeout expires and process is still
-  alive.
+  Raised by :meth:`Process.wait` method if timeout expires and the process is
+  still alive.
+  *name* attribute is available if :meth:`Process.name()` was previously called.
 
 Process class
 -------------
@@ -1445,9 +1453,16 @@ Process class
 
   .. method:: threads()
 
-    Return threads opened by process as a list of named tuples including thread
-    id and thread CPU times (user/system). On OpenBSD this method requires
-    root privileges.
+    Return threads opened by process as a list of named tuples. On OpenBSD this
+    method requires root privileges.
+
+    - **id**: the native thread ID assigned by the kernel. If :attr:`pid` refers
+      to the current process, this matches the
+      `native_id <https://docs.python.org/3/library/threading.html#threading.Thread.native_id>`__
+      attribute of the `threading.Thread`_ class, and can be used to reference
+      individual Python threads running within your own Python app.
+    - **user_time**: time spent in user mode.
+    - **system_time**: time spent in kernel mode.
 
   .. method:: cpu_times()
 
@@ -2552,6 +2567,7 @@ If you want to develop psutil take a look at the `development guide`_.
 Platforms support history
 =========================
 
+* psutil 5.8.1 (2021-10): **MidnightBSD**
 * psutil 5.8.0 (2020-12): **PyPy 2** on Windows
 * psutil 5.7.1 (2020-07): **Windows Nano**
 * psutil 5.7.0 (2020-02): drop Windows XP & Server 2003 support
@@ -2959,6 +2975,7 @@ Timeline
 .. _`subprocess.Popen`: https://docs.python.org/3/library/subprocess.html#subprocess.Popen
 .. _`temperatures.py`: https://github.com/giampaolo/psutil/blob/master/scripts/temperatures.py
 .. _`TerminateProcess`: https://docs.microsoft.com/en-us/windows/desktop/api/processthreadsapi/nf-processthreadsapi-terminateprocess
+.. _`threading.Thread`: https://docs.python.org/3/library/threading.html#threading.Thread
 .. _Tidelift security contact: https://tidelift.com/security
 .. _Tidelift Subscription: https://tidelift.com/subscription/pkg/pypi-psutil?utm_source=pypi-psutil&utm_medium=referral&utm_campaign=readme
 .. _Tidelift Subscription: https://tidelift.com/subscription/pkg/pypi-psutil?utm_source=pypi-psutil&utm_medium=referral&utm_campaign=readme

psutil/__init__.py

@@ -269,7 +269,11 @@ def _assert_pid_not_reused(fun):
     @functools.wraps(fun)
     def wrapper(self, *args, **kwargs):
         if not self.is_running():
-            raise NoSuchProcess(self.pid, self._name)
+            if self._pid_reused:
+                msg = "process no longer exists and its PID has been reused"
+            else:
+                msg = None
+            raise NoSuchProcess(self.pid, self._name, msg=msg)
         return fun(self, *args, **kwargs)
     return wrapper
 
@@ -340,6 +344,7 @@ def _init(self, pid, _ignore_nsp=False):
         self._exe = None
         self._create_time = None
         self._gone = False
+        self._pid_reused = False
         self._hash = None
         self._lock = threading.RLock()
         # used for caching on Windows only (on POSIX ppid may change)
@@ -364,8 +369,7 @@ def _init(self, pid, _ignore_nsp=False):
             pass
         except NoSuchProcess:
             if not _ignore_nsp:
-                msg = 'no process found with pid %s' % pid
-                raise NoSuchProcess(pid, None, msg)
+                raise NoSuchProcess(pid, msg='process PID not found')
             else:
                 self._gone = True
         # This pair is supposed to indentify a Process instance
@@ -571,15 +575,16 @@ def is_running(self):
         It also checks if PID has been reused by another process in
         which case return False.
         """
-        if self._gone:
+        if self._gone or self._pid_reused:
             return False
         try:
             # Checking if PID is alive is not enough as the PID might
             # have been reused by another process: we also want to
             # verify process identity.
             # Process identity / uniqueness over time is guaranteed by
             # (PID + creation time) and that is verified in __eq__.
-            return self == Process(self.pid)
+            self._pid_reused = self != Process(self.pid)
+            return not self._pid_reused
         except ZombieProcess:
             # We should never get here as it's already handled in
             # Process.__init__; here just for extra safety.
@@ -1387,7 +1392,6 @@ def pid_exists(pid):
 
 
 _pmap = {}
-_lock = threading.Lock()
 
 
 def process_iter(attrs=None, ad_value=None):
@@ -1411,58 +1415,59 @@ def process_iter(attrs=None, ad_value=None):
     If *attrs* is an empty list it will retrieve all process info
     (slow).
     """
+    global _pmap
+
     def add(pid):
         proc = Process(pid)
         if attrs is not None:
             proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
-        with _lock:
-            _pmap[proc.pid] = proc
+        pmap[proc.pid] = proc
         return proc
 
     def remove(pid):
-        with _lock:
-            _pmap.pop(pid, None)
+        pmap.pop(pid, None)
 
+    pmap = _pmap.copy()
     a = set(pids())
-    b = set(_pmap.keys())
+    b = set(pmap.keys())
     new_pids = a - b
     gone_pids = b - a
     for pid in gone_pids:
         remove(pid)
-
-    with _lock:
-        ls = sorted(list(_pmap.items()) +
-                    list(dict.fromkeys(new_pids).items()))
-
-    for pid, proc in ls:
-        try:
-            if proc is None:  # new process
-                yield add(pid)
-            else:
-                # use is_running() to check whether PID has been reused by
-                # another process in which case yield a new Process instance
-                if proc.is_running():
-                    if attrs is not None:
-                        proc.info = proc.as_dict(
-                            attrs=attrs, ad_value=ad_value)
-                    yield proc
-                else:
+    try:
+        ls = sorted(list(pmap.items()) + list(dict.fromkeys(new_pids).items()))
+        for pid, proc in ls:
+            try:
+                if proc is None:  # new process
                     yield add(pid)
-        except NoSuchProcess:
-            remove(pid)
-        except AccessDenied:
-            # Process creation time can't be determined hence there's
-            # no way to tell whether the pid of the cached process
-            # has been reused. Just return the cached version.
-            if proc is None and pid in _pmap:
-                try:
-                    yield _pmap[pid]
-                except KeyError:
-                    # If we get here it is likely that 2 threads were
-                    # using process_iter().
-                    pass
-            else:
-                raise
+                else:
+                    # use is_running() to check whether PID has been
+                    # reused by another process in which case yield a
+                    # new Process instance
+                    if proc.is_running():
+                        if attrs is not None:
+                            proc.info = proc.as_dict(
+                                attrs=attrs, ad_value=ad_value)
+                        yield proc
+                    else:
+                        yield add(pid)
+            except NoSuchProcess:
+                remove(pid)
+            except AccessDenied:
+                # Process creation time can't be determined hence there's
+                # no way to tell whether the pid of the cached process
+                # has been reused. Just return the cached version.
+                if proc is None and pid in pmap:
+                    try:
+                        yield pmap[pid]
+                    except KeyError:
+                        # If we get here it is likely that 2 threads were
+                        # using process_iter().
+                        pass
+                else:
+                    raise
+    finally:
+        _pmap = pmap
 
 
 def wait_procs(procs, timeout=None, callback=None):