Home

Welcome to the one-workflow-many-ways wiki!

As the README says, the point of this project is to get an idea of how easy or hard it is for a beginner to implement a basic workflow in different workflow systems. With the exception of bash, I am a beginner at all of these.

bash

I whipped up this script in approximately 10 minutes and then spent another 30 minutes making it nice. I consider this the 'baseline' by which other scripts are measured.

• bash file : https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/bash/bamqc.sh
• results : https://travis-ci.org/morgantaschuk/one-workflow-many-ways/jobs/349431822

WDL

This was the first new workflow language I tried.

• wdl file: https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/wdl/bamqc.wdl
• inputs: https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/wdl/bamqc_inputs.json
• results: https://travis-ci.org/morgantaschuk/one-workflow-many-ways/jobs/349431820

Thoughts:

• The documentation is very good and all in one place. I had a practical working example very quickly
• The WDL file almost like bash.
  • some escaping problems. basically nothing appreciates samtools flagstat $BAM 2>&1 | perl -pe 's|(\d+ \+ \d+)\s+(.*)\R|"$2": "$1",|g' | sed 's/.$//'
  • was finicky about colliding names (can't have a global bamqc variable and a bamqc task)
  • had it working pretty quickly
• Cromwell is a little bit verbose (but this is tunable)
• WOMtools lets you autogenerate the inputs.json file.

CWL

This was a totally different experience.

• cwl files: https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/cwl/workflow.cwl
  • https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/cwl/flagstat.cwl
  • https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/cwl/flagstat2json.cwl
  • https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/cwl/bamqc.cwl
• input file: https://github.com/morgantaschuk/one-workflow-many-ways/blob/master/cwl/workflow.yml
• results: https://travis-ci.org/morgantaschuk/one-workflow-many-ways/jobs/349431821

Thoughts:

• No pipes??? NO PIPES???
  • No you can have pipes but you need to have requirements: class: ShellCommandRequirement and then {valueFrom: " | ", shellQuote: false} (see bamqc.cwl)
  • ..... ok
• Every step in a different cwl file, joined together with a workflow cwl
• documentation is disorganized and the 'user guide' doesn't actually show you anything useful for a really long time http://www.commonwl.org/user_guide/
  • Creating workflows (with multiple steps... remember you can't pipe so each step is very small) is buried down in lesson 20
• Specifying inputs and outputs is a bit bonkers
  • this is how you name output files:

stdout: bamqc_result.json
outputs:
  outjson:
    type: stdout

• yes that means the name is 'out of scope' of the actual outfile. For some reason.
• once I had the separate steps (I re-wrote flagstat2json.sh so that it could take a file as well as stdin) then creating the workflow was very simple 👍 for reuse