ProcessPool hangs when pickling fails

[csm10495]

Hey folks,

I have a case in my code where I was trying to figure out why it seemed to be hanging while waiting for some futures to complete.
It turns out that this seems to happen when the object isn't pickleable (and i guess fork isn't used as the multiprocessing mechanism).

Here is a simple repro:

Python 3.10, on Mac. Though I think it should reproduce on Windows and on Linux if fork is disabled.
Pebble: 5.1.0

from pebble import ProcessPool
from concurrent.futures import ProcessPoolExecutor
import traceback

class TestProcessor:
    def __init__(self):
        # I know this won't pickle well, but that is on purpose to simulate the hang.
        self._my_lambda = lambda: 10

    def task_runner(self, task):
        print(f"task_runner: {task}")
        return f'data: {task}'

    def process_via_pebble(self):
        pool = ProcessPool(max_workers=4)

        futures = set()

        futures.add(pool.submit(self.task_runner, 'task1', timeout=5))
        futures.add(pool.submit(self.task_runner, 'task2', timeout=5))

        for future in futures:
            print(future.result())

    def process_via_native(self):
        pool = ProcessPoolExecutor(max_workers=4)

        futures = set()

        futures.add(pool.submit(self.task_runner, 'task1'))
        futures.add(pool.submit(self.task_runner, 'task2'))

        for future in futures:
            print(future.result())


if __name__ == '__main__':
    f = TestProcessor()

    print("-------------- Native --------------")
    try:
        f.process_via_native()
    except Exception:
        print(".. Handling Exception:")
        traceback.print_exc()

    print("-------------- Pebble --------------")

    try:
        f.process_via_pebble()
    except Exception:
        print(".. Handling Exception:")
        traceback.print_exc()

    print(" .. We don't get here since it hangs")

Running that yields output like so:

-------------- Native --------------
.. Handling Exception:
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'TestProcessor.__init__.<locals>.<lambda>'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/cmachalo/mp/lipy-impact-processing/lipy-impact-processing/playground.py", line 42, in <module>
    f.process_via_native()
  File "/Users/cmachalo/mp/lipy-impact-processing/lipy-impact-processing/playground.py", line 34, in process_via_native
    print(future.result())
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'TestProcessor.__init__.<locals>.<lambda>'
-------------- Pebble --------------
Exception in thread Thread-2 (task_scheduler_loop):
Traceback (most recent call last):
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/cmachalo/mp/lipy-impact-processing/build/lipy-impact-processing/environments/development-venv/lib/python3.10/site-packages/pebble/pool/process.py", line 170, in task_scheduler_loop
    pool_manager.schedule(task)
  File "/Users/cmachalo/mp/lipy-impact-processing/build/lipy-impact-processing/environments/development-venv/lib/python3.10/site-packages/pebble/pool/process.py", line 219, in schedule
    self.worker_manager.dispatch(task)
  File "/Users/cmachalo/mp/lipy-impact-processing/build/lipy-impact-processing/environments/development-venv/lib/python3.10/site-packages/pebble/pool/process.py", line 354, in dispatch
    self.pool_channel.send(WorkerTask(task.id, task.payload))
  File "/Users/cmachalo/mp/lipy-impact-processing/build/lipy-impact-processing/environments/development-venv/lib/python3.10/site-packages/pebble/pool/channel.py", line 70, in send
    return self.writer.send(obj)
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/System/Volumes/Data/export/apps/python/3.10.14/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'TestProcessor.__init__.<locals>.<lambda>'

and then it hangs right there.

If the object/method wasn't pickleable, it should raise similar to how the native ProcessPoolExecutor would so that it doesn't continue and then hang on .result()

[noxdafox]

Issue addressed, it will be fixed within the next release. Thanks for reporting!

[csm10495]

Thanks!

[noxdafox]

I usually close the issue on release. 😄