Consider implementing the Common Workflow Language standard #873

mr-c · 2018-06-02T07:41:04Z

https://www.commonwl.org

zanieb · 2019-07-31T16:42:29Z

+1

Would be willing to contribute to this.

cuericlee · 2019-11-01T06:51:01Z

maybe we could consider to parse both wdl/cwl into DAG and parse inputs as parameters in argo, take NAS volume as an example PoC, then go object storage.

https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.wdl
https://github.com/gatk-workflows/broad-prod-wgs-germline-snps-indels/blob/master/JointGenotypingWf.hg38.inputs.json

alexec · 2020-05-13T15:40:11Z

This is unlikely to be developed by the core team at this time - it is a perfect piece of work for community contribution.

alexec · 2021-08-26T17:37:55Z

I wonder if a transpiler from CWS to Argo YAML would solve this.

isubasinghe · 2021-12-07T01:08:16Z

Picked this up, going to be developing on a CWL to argo transpiler on my fork https://github.com/isubasinghe/argo-workflows

isubasinghe · 2021-12-09T22:40:14Z

Hey @mr-c would you by any chance have time for a quick chat about CWL? Perhaps we could setup a small zoom meeting?

isubasinghe · 2021-12-09T23:38:30Z

Thanks I have booked a meeting for Monday morning, looking forward to chatting with you

) * feat: Implemented Exact Once triggering for NATS event bus * update comments * still comments * fix doc format * fix typo * lowercase id * correct start position * ackwait change back to 1 sec * log minor change

alexec · 2022-03-09T21:05:25Z

See #7503, #7506, #7538, #7553 ,#7578

isubasinghe · 2022-03-09T21:16:46Z

Apologies for not getting back on to this, my latest work is available on my branch here: https://github.com/isubasinghe/argo-workflows/tree/feat/cwl2argo-output/cmd/cwl2argo

It is capable of compiling "CommandLineTool" types but it would be trivial to reuse the functions to make the compiler work for "Workflow" types.

Unfortunately I am busy with university and settling in another country at the moment so I haven't had time to work on Argo Workflows.

alexec · 2022-03-09T21:29:57Z

Would anyone watching this issue like to pick this up?

SerRichard · 2024-04-22T08:04:13Z

@alexec Hi, I am happy to pick this up, it's come up at work a couple of times over the last year.

Would you consider a PR where this is encapsulated as a new argo cli command? Something like, argo transpile FILE ?

isubasinghe · 2024-04-22T08:55:19Z

Hey @SerRichard, I'd be happy to help you out with this. I have a parser for CWL that would help you.
It already supports some amount of argo.

It's unfortunately not as simple as I personally thought it was at the time.

SerRichard · 2024-04-22T09:03:50Z

@isubasinghe The code in the fork, or something else? & do you remember the sticking points you found when trying to work on it before?

mr-c · 2024-04-22T09:19:11Z

For parsing CWL in Go, I recommend adding Go support to our code generator. It is already used to make CWL parsers for C++, Python, Java, Typescript, .Net, and soon also Rust:

https://schema-salad.readthedocs.io/en/latest/#codegen-examples

(I can provide guidance and mentoring for this task)

isubasinghe · 2024-05-02T10:07:21Z

@SerRichard Yeah the fork has an additional cli, I think one of the branches are pretty much complete for a CWL type as well.

Probably not much for you to do from a parser and type checking point of view.
I can't remember the exact pain points, but generally speaking the correspondence between CWL and argo isn't one to one and requires some creativity to address.

My tool already does some generation, this: https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/argo_simple.yaml was generated from https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/locations.json and https://github.com/isubasinghe/argo-workflows/blob/feat/cwl2argo-output/cmd/cwl2argo/hello.cwl

SerRichard · 2024-05-20T11:39:00Z

@isubasinghe @mr-c Sorry for the delay replying, I've only now had a chance to look at this again. Going through the cwl spec there's a couple of places where I'm not certain what the expected behaviour of a cwl2argo parser would be.

Firstly, take the example of a CommandLineTool where no docker requirement is defined, or only software requirements are defined. Install commands could be added to a step's script in a workflow, but in both cases I would still expect the user to define some base environment that this cli is expected to run in. So could Docker requirements with a docker pull be a requirement for the parser? If not, do you have any other ideas on how to handle this?

Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.

Any input is appreciated!

mr-c · 2024-05-20T19:00:09Z

So could Docker requirements with a docker pull be a requirement for the parser?

Yes, that is reasonable, and true for basically all the cloud-based CWL compatible platforms 👍

Secondly, I don't totally grasp how to map cwl mounts to either argo workflow mounts or artifacts. For example, when a Dirent in cwl is specified, the files must be staged before execution. If that is the case, then the argo workflow should reference either a mount or artifact. Is there a way to infer from cwl which syntax ought to be used when parsing to an argo worflow? If not, then I'm not sure how to get away from asking a user to parse a flag when calling the cwl2argo parser.

I'm not clear on the question here. Can you share some examples?

SerRichard · 2024-07-31T09:15:23Z

Hi again, sorry for the delay! On my fork I've taken the initial implementation from @isubasinghe and extended to also work for the CWL Workflow, currently I'm still missing some handling ( e.g. scatter & scatter method, and other bits ), I need to write a set of unit tests, and I'll need to tidy up the branch quite a bit, but the basic parsing for a CWL Workflow is there.

@alexec I'd like to open a draft PR, in order to get some feedback and continue the work with some general direction.

@mr-c forget my earlier question, I was a bit muddled about when to use an artefact vs output paramter when transpiling to an argo workflow. Though, I think that this can either be set explicitly, or inferred from the class type of the CWL provided. Additionally, the type checking has grown quite a lot, and will continue to grow if I were to add support for Operations and ExpressionTools. I might be able to work on your earlier mention of Go support for parsing CWL, if you like we can set up a meeting and discuss that in parrallel?

agilgur5 · 2024-08-03T19:47:09Z

Curious, is there any reason for this tool to exist as part of the core Argo repo? It sounds like it would work perfectly fine as its own separate tool, no?

We can certainly host it in argoproj-labs as well

mr-c · 2024-08-04T09:02:19Z

@mr-c I might be able to work on your earlier mention of Go support for parsing CWL, if you like we can set up a meeting and discuss that in parrallel?

Happily! My email username is mrc and the domain name is commonwl.org

Here's a video from this year's CWL conference showing how they did the same for Rust: https://cwl.discourse.group/t/salad-goes-rusty-expanding-the-horizons-of-cwl/907

SerRichard · 2024-08-19T06:42:06Z

@agilgur5 I was thinking the same while I was working on this ( albeit not specifically where the tool would live ), though @isubasinghe might know of a reason why this was planned to be included in the core repo? I'm happy to continue the work as part of a tool in the argoproj-labs project, for my own use cases this might even be preferable.

isubasinghe · 2024-08-19T06:50:36Z

@SerRichard no real reason that it should exist in this repo.
It would be better served in argoproj-labs.

It was early days into my argo workflows contribution that I worked on this and just didn't know about the existence of argoproj-labs.

shuangkun · 2024-08-19T13:21:36Z

Can we support WDL too? We have encountered many wdl workflows running on cromwell that want to run on kubernetes.

agilgur5 · 2024-08-19T15:24:53Z

I don't see why not, anything useful can be hosted in argoproj-labs. That may very well need its own contributor to champion though

SerRichard · 2024-08-20T11:35:56Z

@agilgur5 How would you advise we go about starting this? I can isolate existing code into an intial poc, should this be done on my own git and we can move it later, or would you create a project in argoproj-labs?

isubasinghe · 2024-08-20T11:38:12Z

@SerRichard could we review this on your own git repo first before we create a repo for it?
I'd like to take a look first.

SerRichard · 2024-08-20T11:56:24Z

@isubasinghe Yeah ofc, I'll work the fork into a tool and link back here for review!

agilgur5 · 2024-08-20T13:41:16Z

Usually labs projects are transferred from personal repos into the org and have at least 1 Argoproj maintainer sponsor. It's not a completely robust process but here's an example issue: argoproj-labs/community#11

shuangkun · 2024-09-05T02:05:16Z

@SerRichard I am also interested in this project.

shuangkun · 2024-09-18T03:26:17Z

I sorted out some previous work and opened an issue argoproj-labs/community#13

SerRichard · 2024-10-03T07:34:11Z

Apologies for the silence, I've been carried away with work. I'd put the previous work into a project here Proteus, @shuangkun mb we can look at merging the two projects? I'm not sure if the aim should be merging, or having seperate tooling for wdl, cwl, and any other possible additions.

@agilgur5 I'd appreciate some review on the code there, there's a couple of bugs I'm aware of that I need to resolve, but it's also my first project in Go, so I'm sure there are some glaring mistakes that I've missed.

isubasinghe · 2024-10-04T00:48:44Z

I think I'd prefer to merge the two projects really, they would both have some code they would need to share, but more importantly it reduces the number of projects with introduce to argoproj-labs.

But thanks a lot @SerRichard I will have a look.

mr-c · 2024-10-22T13:09:46Z

@SerRichard @isubasinghe any updates on the CWL side? I'm happy to review the work, provide feedback, etc..

isubasinghe · 2024-10-23T02:21:08Z

I haven't reviewed the work yet, been quite busy. Any help reviewing would significantly speed up the process to submitting a request for argoproj-labs @mr-c

isubasinghe · 2024-10-30T22:39:17Z

@SerRichard hope you aren't disheartened by the lack of progress on this, I promise you this is on my todo. I'm hoping I get a bit more availability after kubecon.