feat: CLIN-3267 draft

[LysianeBouchard]

WORK IN PROGRESS

I am opening this pull request for transparency and welcome any feedback. However, please note that this work is not final and not ready to be merged.

DESCRIPTION

This PR introduces the etl variant annotation dag. It also restructure the nextflow operator to mimick the structure in cqdg, i.e. we introduce similar base config / operator classes.

So far, this code has been tested locally with Minikube for test_nextflow_operator and etl_variant_annotation. It has not yet been tested on etl.py.

There are a few TODOs in the code that need to be addressed, especially regarding the variant annotation DAG.

CQDG ALIGNMENT
This implementation reuses a configuration mechanism similar to CQDG, with the following key differences:

• 2 different k8s contexts: In this repo, there are 2 different k8s contexts: ETL and DEFAULT. We assume that we will create 2 different instances of KubeConfig, one for each context. So far only the ETL context KubeConfig instance is created.
• service_account_name specified in BaseConfig instead of KubeConfig: This is because nextflow and spark operators have different service account names. I thought the intention of the KubeConfig class is to capture what is always the same and thought that, since it can vary, perhaps it should go in BaseConfig instead. There are other solutions of course. Let me know if you are ok with this one.
• shallow copy instead deep copy: In BaseConfig, shallow copy is used instead deep copy in build_operator and partial methods. This is to avoid serializing more complex objects as dictionaries.
• BaseConfig.args renamed to BaseConfig.append_args: I suspect that one might think that args overwrite the arguments instead appending, but well this is subjective.

NEXTFLOW OPERATOR
The nextflow operator now inherits the BaseKubernetesOperator . It is also modified to support multiple config maps.

VARIANT ANNOTATION
We introduce a new variant annotation dag. This part needs a bit more work.