Let's not use Python 3.9.0 anywhere

[jluethi]

When using Python 3.9.0 to run the server, I keep running into issues like this one with tasks like scmultiplex and the 2D to 3D tasks:

I don't get those errors when the server is running on a more recent Python version (=> the tasks then also use that more recent Python version).

Thus, let's make sure we don't use any Python 3.9.0 anywhere for the server. I changed the install scripts to use Python 3.10, this issue is just to document the reasoning behind.

TASK ERROR:Task id: 128 (Convert 2D Segmentation to 3D), e.workflow_task_order=6
TRACEBACK:
2023-09-13 22:18:08,860; INFO; START convert_2D_segmentation_to_3D task
Traceback (most recent call last):
  File "/Users/joel/Library/CloudStorage/Dropbox/Joel/BioVisionCenter/Code/fractal/napari-ome-zarr-roi-loader/examples/convert_2D_segmentation_to_3D.py", line 161, in <module>
    run_fractal_task(
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/fractal_tasks_core/tasks/_utils.py", line 79, in run_fractal_task
    metadata_update = task_function(**pars)
  File "pydantic/decorator.py", line 40, in pydantic.decorator.validate_arguments.validate.wrapper_function
  File "pydantic/decorator.py", line 134, in pydantic.decorator.ValidatedFunction.call
  File "pydantic/decorator.py", line 206, in pydantic.decorator.ValidatedFunction.execute
  File "/Users/joel/Library/CloudStorage/Dropbox/Joel/BioVisionCenter/Code/fractal/napari-ome-zarr-roi-loader/examples/convert_2D_segmentation_to_3D.py", line 108, in convert_2D_segmentation_to_3D
    write_labels(
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/ome_zarr/writer.py", line 823, in write_labels
    dask_delayed_jobs = _write_dask_image(
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/ome_zarr/writer.py", line 594, in _write_dask_image
    da.compute(*delayed)
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/dask/base.py", line 615, in compute
    schedule = get_scheduler(
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/dask/base.py", line 1440, in get_scheduler
    from distributed import get_client
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/distributed/__init__.py", line 23, in <module>
    from distributed.actor import Actor, ActorFuture, BaseActorFuture
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/distributed/actor.py", line 13, in <module>
    from distributed.client import Future
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/distributed/client.py", line 115, in <module>
    from distributed.worker import get_client, get_worker, secede
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/distributed/worker.py", line 117, in <module>
    from distributed.worker_memory import (
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/site-packages/distributed/worker_memory.py", line 55, in <module>
    WorkerDataParameter: TypeAlias = Union[
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/typing.py", line 243, in inner
    return func(*args, **kwds)
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/typing.py", line 316, in __getitem__
    return self._getitem(self, parameters)
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/typing.py", line 421, in Union
    parameters = _remove_dups_flatten(parameters)
  File "/Users/joel/mambaforge/envs/fractal-client-1.3.4a1/lib/python3.9/typing.py", line 215, in _remove_dups_flatten
    all_params = set(params)
TypeError: unhashable type: 'list'

[tcompa]

This is annoying, but I think the problem is with the tasks (and then it "propagates" to the server, because by default we are using the same python version if not specified).

Two comments:

• One should try not to use "old" versions of python 3.9 (this issue reportedly affects 3.9.0 and 3.9.1 but not 3.9.2: Error importing distributed with python=3.9.0 and 3.9.1 dask/distributed#7956 - see also Test suite failure for commit  ome/ome-zarr-py#286);
• Something should fail upfront, and not while executing tasks. If we can reproduce it in a controlled scenario, and verify that only 3.9.0 and 3.9.1, then I'd be in favor of making it a hard requirement for fractal-tasks-core.

EDIT: this is actually an ome-zarr-py error, which is triggered by scmultiplex. Therefore I cannot say (yet) whether it also affects fractal-tasks-core.

[jluethi]

In my testing, the fractal-tasks-core tasks were running fine, but the scmultiplex & 2D to 3D tasks ran into issues. 2D to 3D use the ome-zarr-py package, but scmultiplex doesn't have such a dependency. The error above is from 2D to 3D. scmultiplex runs into a similar error though.

If it really is related to Python 3.9.0 / .1 issues, then the easiest way around for the moment is just not to default to use those. Thus either picking a newer 3.9 distro or (my suggestion) switching our templates to using 3.10 distributions.
Having switched the UZH server to 3.10, I can't reproduce this behavior anymore

[tcompa]

In my testing, the fractal-tasks-core tasks were running fine, but the scmultiplex & 2D to 3D tasks ran into issues. 2D to 3D use the ome-zarr-py package, but scmultiplex doesn't have such a dependency. The error above is from 2D to 3D. scmultiplex runs into a similar error though.

The error in ome-zarr-py comes from a dask.distributed error (see dask/distributed#7956) and maybe from somewhere even more upstream (the dask issue mentions a Python bug, likely this one, which was then fixed in 3.9.2). Thus it's not surprising that you can hit it without depending on ome-zarr-py.