Update taskflow and iRODS functionality for iRODS 4.3

[mikkonie]

Related to bihealth/irods-docker#16, bihealth/sodar-docker-compose#34 and bihealth/davrods-docker#3.

I have been working on updating our iRODS docker setup from 4.2 to 4.3. At the moment the server runs ok, but taskflow operations from SODAR are failing. I'll add an error dump below.

I'll be looking into fixing this and updating sodar-docker-compose for SODAR v0.14.0, as I really want to get the current release out along with our manuscript resubmission.

It's possible the python-irodsclient is simply too old, but upgrading that will create a conflict with cubi-tk..

This will most likely break compatibility with iRODS 4.2 and below.

[mikkonie]

Error in SODAR:

2023-02-01 09:14:57,898 [INFO] taskflowbackend.api: Building flow "project_create"..
2023-02-01 09:14:57,900 [INFO] taskflowbackend.api: Building flow OK
2023-02-01 09:14:57,900 [INFO] taskflowbackend.flows: Running flow "project_create"
2023-02-01 09:14:57,940 [INFO] taskflowbackend.tasks: Executed: <iRODS> Create root collection for SODAR projects (CreateCollectionTask)
2023-02-01 09:14:57,973 [INFO] taskflowbackend.tasks: Executed: <iRODS> Create collection for project (CreateCollectionTask)
2023-02-01 09:14:57,980 [INFO] taskflowbackend.tasks: Executed: <iRODS> Add title metadata to project (SetCollectionMetadataTask)
2023-02-01 09:14:57,982 [INFO] taskflowbackend.tasks: Reverted: <iRODS> Add title metadata to project (SetCollectionMetadataTask)
2023-02-01 09:14:58,045 [INFO] taskflowbackend.tasks: Reverted: <iRODS> Create collection for project (CreateCollectionTask)
2023-02-01 09:14:58,073 [INFO] taskflowbackend.tasks: Reverted: <iRODS> Create root collection for SODAR projects (CreateCollectionTask)
2023-02-01 09:14:58,074 [ERROR] taskflowbackend.flows: Exception in run_flow(): None
2023-02-01 09:14:58,075 [ERROR] taskflowbackend.api: Error running flow: None
2023-02-01 09:14:58,075 [ERROR] projectroles.views: Exception in perform_project_modify() for plugin "taskflow": Error running flow: None
Internal Server Error: /project/create/eddbfd99-2d81-4b21-ab02-e4455f3fcca9
Traceback (most recent call last):
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 143, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/contrib/auth/mixins.py", line 104, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/views/generic/edit.py", line 172, in post
    return super().post(request, *args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar12/lib/python3.8/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 1196, in form_valid
    raise ex
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 1174, in form_valid
    project = self.modify_project(
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 1131, in modify_project
    self.call_project_modify_api(
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 816, in call_project_modify_api
    raise ex
  File "/home/mikkopen/.virtualenvs/sodar12/src/django-sodar-core/projectroles/views.py", line 798, in call_project_modify_api
    getattr(p, method_name)(*method_args)
  File "/home/mikkopen/code/sodar/taskflowbackend/plugins.py", line 115, in perform_project_modify
    taskflow.submit(
  File "/home/mikkopen/code/sodar/taskflowbackend/api.py", line 244, in submit
    return self.run_flow(
  File "/home/mikkopen/code/sodar/taskflowbackend/api.py", line 187, in run_flow
    cls._raise_flow_exception(ex_msg, tl_event, None)
  File "/home/mikkopen/code/sodar/taskflowbackend/api.py", line 60, in _raise_flow_exception
    raise cls.FlowSubmitException(ex_msg)
taskflowbackend.api.TaskflowAPI.FlowSubmitException: Error running flow: None

[mikkonie]

Creating landing zones currently fails with "create collections" enabled. I suspect this may have something to do with ACL changes in iRODS 4.3. To be investigated. Dump:

[2023-10-11 16:04:07,998: ERROR/ForkPoolWorker-8] Task taskflowbackend.tasks_celery.submit_flow_task[be644992-1625-4f8b-a763-c63ad87724ac] raised unexpected: FlowSubmitException("Error running flow: 'read_object'")
Traceback (most recent call last):
  File "/home/mikkopen/.virtualenvs/sodar14/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/mikkopen/.virtualenvs/sodar14/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/mikkopen/code/sodar/taskflowbackend/tasks_celery.py", line 29, in submit_flow_task
    taskflow.run_flow(
  File "/home/mikkopen/code/sodar/taskflowbackend/api.py", line 190, in run_flow
    cls._raise_flow_exception(ex_msg, tl_event, None)
  File "/home/mikkopen/code/sodar/taskflowbackend/api.py", line 63, in _raise_flow_exception
    raise cls.FlowSubmitException(ex_msg)
taskflowbackend.api.TaskflowAPI.FlowSubmitException: Error running flow: 'read_object'

[mikkonie]

The aforementioned issue seems to stem from the ACL strings returned being in a different format:

======================================================================
FAIL: test_execute_write (taskflowbackend.tests.test_tasks.TestSetDataObjAccessTask)
Test access setting for write/modify
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mikkopen/code/sodar/taskflowbackend/tests/test_tasks.py", line 1305, in test_execute_write
    self.assertEqual(user_access.access_name, TEST_ACCESS_WRITE_OUT)
AssertionError: 'modify_object' != 'modify object'
- modify_object
?       ^
+ modify object
?       ^

[mikkonie]

Done. The API operation between 4.2 and 4.3 only differs by a couple of API parameters, so I was able to make the change in a non-breaking way. Hence, the current SODAR version works identically on both iRODS 4.2 and 4.3. I'll add separate issues for setting the default version and removing 4.2 support.