Running the tests raise "TypeError: cannot pickle '_io.TextIOWrapper' object"

[amotl]

Hi there,

I am trying to give this repository and the test suite some love on behalf of #397 and I am again tripped by this error when running the test suite partially on macOS, like

Error in test test_client_keepalive (crate.client.test_http.KeepAliveClientTest)
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 589, in run
    self._callSetUp()
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 547, in _callSetUp
    self.setUp()
  File "/Users/amo/dev/crate/sources/crate-python/src/crate/client/test_http.py", line 404, in setUp
    self.server_process.start()
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object

I already touched that at #386 (comment).

With kind regards,
Andreas.

[amotl]

Others observed the same thing and reported about it at [1,2,3]. Apparently, this error comes from a change in Python 3.8 in the multiprocessing library, as reported at GoogleCloudPlatform/gsutil#961 (comment).

In this context, I also found [4] and [5].

The solution, according to GoogleCloudPlatform/gsutil#1107, is to explicitly set the multiprocessing start method to 'fork'. Apparently, this is not the default for all platforms starting with Python 3.8.

[1] https://stackoverflow.com/questions/62903886/why-is-giving-me-this-error-typeerror-cannot-pickle-io-textiowrapper-object
[2] https://0xbanana.com/post/gsutil-typeerror-cannot-pickle-_io.textiowrapper-object/
[3] GoogleCloudPlatform/gsutil#961
[4] https://bugs.python.org/issue33725
[5] https://bugs.python.org/issue33884