test_pre_import fails in some environments

[pentschev]

In some environments test_pre_import may fail as the module that is picked arbitrarily may not be importable by the dask cuda worker process. This behavior has been observed in CI when testing wheels, but testing conda doesn't present any problems at this time, but given the non-deterministic nature of the test it may start failing at some point. The traceback of the error observed is as follows:

Traceback (most recent call last):
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/core.py", line 528, in start
    await wait_for(self.start_unsafe(), timeout=timeout)
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/utils.py", line 1915, in wait_for
    return await asyncio.wait_for(fut, timeout)
  File "/pyenv/versions/3.10.16/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/worker.py", line 1493, in start_unsafe
    raise plugins_exceptions[0]
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/utils.py", line 805, in wrapper
    return await func(*args, **kwargs)
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/distributed/worker.py", line 1890, in plugin_add
    result = plugin.setup(worker=self)
  File "/pyenv/versions/3.10.16/lib/python3.10/site-packages/dask_cuda/plugins.py", line 213, in setup
    importlib.import_module(l)
  File "/pyenv/versions/3.10.16/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'benchmarks'

Ideally, we would pick a module that we are absolutely certain is not gonna be loaded when pre-import runs, but we don't control the entire software stack to be always sure this is going to be the case for any module we choose. To test LocalCUDACluster there would be a simple solution to that, we could register a module at runtime as below:

import types
import sys

mock = types.ModuleType('mock')

# Add it to sys.modules
sys.modules['mock'] = mock

import mock

We could then simply try to import the module as above. However, with dask cuda worker that's not possible, given the process we run with popen will not inherit anything we do at runtime. One alternative I can think is perhaps creating a submodule like dask_cuda/utils_test_mock.py we can control not to be loaded, it feels a bit dirty but may do the trick.