-
Notifications
You must be signed in to change notification settings - Fork 306
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: updates
README.md with new information, deduplicates content
- Loading branch information
1 parent
37c93f4
commit 9acbffa
Showing
2 changed files
with
99 additions
and
124 deletions.
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,151 +1,126 @@ | ||
| # Workflow Description Language (WDL) | ||
| <div style="align: center"> | ||
| <img src="https://raw.githubusercontent.com/claymcleod/wdl/refs/heads/docs/readme-update/.github/assets/readme-header.png"> | ||
| </div> | ||
|
|
||
| <hr /> | ||
|
|
||
| The **Workflow Description Language (WDL)** (pronounced as _/hwɪdl/_ or | ||
| _"wh-ih-d-l"_—"whittle" with a 'd') is an open standard for describing both data | ||
| processing workflows using a human-readable/writeable syntax. This domain-specific | ||
| language aims to make it easy to define atomic units of computation (tasks), connect | ||
| them together to achieve a broader data processing goal (workflows), and effortlessly | ||
| scale their execution. | ||
|
|
||
| Fundamentally, the language strives to be accessible and understandable to all manner of | ||
| users, including software engineers, domain experts, and providers of production | ||
| computational systems. Further, the language aims to enable idiomatic workflow execution | ||
| patterns (e.g., conditional execution and scatter-gather constructs) to be simply | ||
| expressed. Last, the language aims to be open and accessible—there is no single | ||
| execution engine, and anyone can develop an execution environment for WDL. Today, | ||
| [several engines and platforms](#execution-engines-and-platforms) exist and cover most | ||
| execution environments (all major HPCs, all major clouds, local execution). | ||
|
|
||
| The **Workflow Description Language (WDL)** is an open standard for describing data processing workflows with a human-readable and writeable syntax. | ||
| WDL makes it straightforward to define analysis tasks, connect them together in workflows, and parallelize their execution. | ||
| The language strives to be accessible and understandable to all manner of users, including programmers, analysts, and operators of a production system. | ||
| The language enables common patterns, such as scatter-gather and conditional execution, to be expressed simply. | ||
| WDL is designed for portability, and there are several [implementations](#execution-engines-and-platforms) to choose from that run in a variety of environments, including HPC systems and cloud platforms. | ||
| ## Getting Started | ||
|
|
||
| ## Versioning | ||
| If you're new to WDL, welcome 👋—we _highly_ recommend you read the [getting started | ||
| documentation][wdl-docs] to learn how to write workflows. Beyond that, here are some | ||
| quickstart guides on various topics regarding getting up to speed with WDL. | ||
|
|
||
| WDL versioning follows [semantic versioning](https://semver.org) conventions. | ||
| ### Learn how to write tasks and workflows | ||
|
|
||
| The WDL *language* has a two-number version (e.g., `1.2`). | ||
| An increase in the minor (second) version number (e.g., `1.1` to `1.2`) indicates the addition of, or non-breaking changes to, the language or standard library functions. | ||
| An increase in the major (first) version number (e.g., `1.0` to `2.0`) indicates that breaking changes have been made. | ||
| **[TODO]** Learn from the of compendium of publicly available workflows and tasks. | ||
|
|
||
| The WDL *specification* has a three-number version (e.g., `1.2.0`). | ||
| The specification version tracks the language version, but there may also be patch releases (indicated by a change to the patch, or third, version number) that include fixes for typos, additional examples, or non-breaking clarifications of ambiguous language. | ||
| ### Running and scaling execution | ||
|
|
||
| ## Language Specifications | ||
| **[TODO]** Explore the execution engines that can run WDL. | ||
|
|
||
| The WDL specification contains all relevant information for users and developers, including those wanting to implement an execution engine. | ||
| This GitHub project uses the branch for the current version of the specification as its primary branch, so you will always see the current version of the specification so long as you visit this project's [root URL](https://github.com/openwdl/wdl). | ||
| Users are strongly encouraged to use the current version of the specification unless absolutely necessary. | ||
| ### Join the community | ||
|
|
||
| This branch is for version **1.2** of the [WDL language specification](https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md). | ||
| All development of new *non-breaking* features should be done against this branch. | ||
| The WDL community depends on your involvement to thrive. You are encouraged to ask | ||
| questions, help other users, and make contributions where you can. Interactions occur | ||
| primarily on [GitHub][github-main] and [Slack][join-slack]. The WDL community also has | ||
| an official [website](https://openwdl.org/) and [blog](https://openwdl.org/blog/) that | ||
| you can follow to stay up to date. | ||
|
|
||
| Previous versions of the spec can be found here: | ||
| * **If you have questions about using WDL,** search in the | ||
| [discussions](https://github.com/openwdl/wdl/discussions) to see if the question has | ||
| been asked already; if not, feel free to start a new discussion. You can also join the | ||
| `#support` channel on [Slack][join-slack] to ask more transient questions. | ||
| * **If you are a user of an execution engine and encounter a bug or want to request a | ||
| feature,** you are encouraged to file an issue on the relevant execution engine's | ||
| repository. Please do not file these types of questions/requests on the main WDL | ||
| specification repository. | ||
| * **If you are an implementor and need clarification on the specification or want to | ||
| request a feature to be added to the WDL language,** please file an issue on the WDL | ||
| specification's [GitHub issues][github-issues] page. | ||
| - **If you'd like to contribute to the WDL specification,** please read the | ||
| [contributing](#contributing) section before submitting a pull request. | ||
|
|
||
| - [1.1](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md) | ||
| - [1.0](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md) | ||
| ### Conclusion | ||
|
|
||
| There are a number of draft versions that correspond to initial efforts at creating WDL. | ||
| While these are functional specifications, they should not be considered feature complete, and they contain many bugs and irregularities. | ||
| If you're interested in diving deeper into the specification itself, feel free to | ||
| continue reading. | ||
|
|
||
| - [draft-3](https://github.com/openwdl/wdl/blob/main/versions/draft-3/SPEC.md) | ||
| - [draft-2](https://github.com/openwdl/wdl/blob/main/versions/draft-2/SPEC.md) | ||
| - [draft-1](https://github.com/openwdl/wdl/blob/main/versions/draft-1/SPEC.md) | ||
| ## Specification | ||
|
|
||
| The next *major* version of the specification is [2.0](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md). | ||
| All development of new *breaking* features should be done against that branch. | ||
| The WDL specification contains all relevant information for users and developers, | ||
| including those wanting to implement an execution engine. This GitHub project uses the | ||
| branch for the current version of the specification as its primary branch, so you will | ||
| always see the current version of the specification so long as you visit this project's | ||
| [root URL][github-main]. Users are strongly encouraged to use the current version of the | ||
| specification unless absolutely necessary. | ||
|
|
||
| ## Community and Support | ||
| * This branch is for version **1.2** of the specification | ||
| ([link](https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md)). All development of new | ||
| *non-breaking* features should be done against this branch. | ||
| * The next major version of the specification is `v2.0` | ||
| ([link](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md)). All development of new | ||
| *breaking* features should be done against that branch. | ||
|
|
||
| The WDL community depends on your involvement to thrive. | ||
| You are encouraged to ask questions, help other users, and make contributions where you can. | ||
| Interactions occur primarily on [GitHub](https://github.com/openwdl/wdl) and [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw). | ||
| The WDL community also has an official [blog](https://dev.to/openwdl) where announcements are made. | ||
|
|
||
| ### Asking a Question | ||
| If desired, previous versions of the spec can be found on the [GitHub branches | ||
| page][github-branches]. | ||
|
|
||
| - Search in the [discussions](https://github.com/openwdl/wdl/discussions) to see if the question has been asked already; if not, start a new discussion. | ||
| - Join our [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) and ask in the `#support` channel. | ||
| - Search the [Bioinformatics Stack Exchange](https://bioinformatics.stackexchange.com/search?q=wdl) for previously answered questions, or ask a new question. | ||
| - Search the [Google Group](https://groups.google.com/a/openwdl.org/forum/#!forum/community) for previously answered questions. This group is largely inactive, so you're encoraged to ask new questions in one of the above places instead. | ||
| ### Versioning | ||
|
|
||
| ### Bugs and New Features | ||
| All products of the WDL project, including the specification, follow [semantic | ||
| versioning](https://semver.org). That being said, the version of the WDL _language_ and | ||
| the version of the WDL _specification_ are distinct: | ||
|
|
||
| - Search for an existing [issue](https://github.com/openwdl/wdl/issues). If your issue has not already been reported, create a new one. | ||
| - For feature reqeusts, you are encoraged to first start a discussion at one of the places listed above to get feedback from the community. | ||
| - If you'd like to provide a fix/implementation for an issue, please read about [contributing](#contributing) before submitting a [pull request](https://github.com/openwdl/wdl/pulls). | ||
| * The WDL _language_ has a two-number version (e.g., `1.2`). An increase in the minor | ||
| (second) version number (e.g., `1.1` to `1.2`) indicates the addition of, or | ||
| non-breaking changes to, the language or standard library functions. An increase in the | ||
| major (first) version number (e.g., `1.0` to `2.0`) indicates that breaking changes have | ||
| been made. | ||
|
|
||
| ### Documentation | ||
| * The WDL _specification_ has a three-number version (e.g., `1.2.0`). The specification | ||
| version tracks the language version, but there may also be patch releases (indicated by | ||
| a change to the patch, or third, version number) that include fixes for typos, | ||
| additional examples, or non-breaking clarifications of ambiguous language. | ||
|
|
||
| - [wdl-docs](https://docs.openwdl.org/en/stable/) | ||
| - [learn-wdl](https://github.com/openwdl/learn-wdl) | ||
| - [WDL Resources](https://support.terra.bio/hc/en-us/sections/360007274612-WDL-Resources) provided by Terra (a product of the Broad Institute) | ||
| ### Contributing | ||
|
|
||
| ## Published Workflows | ||
| WDL only advances through community contributions. While participating in discussions | ||
| and submitting issues are great ways to be involved, help is also needed to implement | ||
| changes to the specification. For more information on how you can contribute, please | ||
| read the [Contributing](CONTRIBUTING.md) guide. | ||
|
|
||
| ### Governance | ||
|
|
||
| The following are collections of open-source WDL workflows. | ||
| The WDL task or workflow you need may already be available in one of these repositories, or you may find a similar workflow and customize it to your needs. | ||
| The WDL specification is entirely community driven; however, it is overseen by a | ||
| [Governance committee](GOVERNANCE.md). If you are interested in being involved in WDL | ||
| governance, please join the [Slack][join-slack] and post a message in the `#general` | ||
| channel. | ||
|
|
||
| - [Dockstore](https://dockstore.org/search?entryType=workflows&descriptorType=WDL&searchMode=files) | ||
| - [BioWDL](https://github.com/biowdl) | ||
| - [Broad Institute WARP](https://broadinstitute.github.io/warp/docs/get-started/) | ||
| - [GATK Workflows](https://github.com/gatk-workflows/) | ||
| - [ENCODE](https://www.encodeproject.org/pipelines/) | ||
| - [Terra](https://app.terra.bio) (requires registration) | ||
| ### License | ||
|
|
||
| ## Software and Tools | ||
| The WDL specification is made available to you under the permissive [BSD 3-Clause | ||
| License](./LICENSE). | ||
|
|
||
| ### Execution Engines and Platforms | ||
| © 2024 The OpenWDL Developers | ||
|
|
||
| WDL does not have an official implementation. | ||
| Third parties are relied upon to provide installable software or hosted platforms that interpret and execute WDL workflows and tasks. | ||
| Although WDL does not yet have an official compliance program or certification process, implementers are expected to design their tools according to the specification so as to maximize the portability of workflows across implementations. | ||
| Nonetheless, implementers may provide additional optional features specific. | ||
| Please see the documentation associated with each tool/platform for information on available execution options and support. | ||
|
|
||
| | Implementation | Requires Installation | Local Execution | HPC | Cloud | | ||
| | ------------------------------------------------------------------------------ | --------------------- | --------------- | ----- | --------------------- | | ||
| | [AWS HealthOmics](https://docs.aws.amazon.com/omics/latest/dev/workflows.html) | No | No | No | AWS | | ||
| | [Cromwell](https://github.com/broadinstitute/cromwell) * | Yes | Yes | Many | AWS Batch, Azure, GCP | | ||
| | [dxCompiler](https://github.com/dnanexus/dxCompiler) | Yes | No | No | DNAnexus | | ||
| | [MiniWDL](https://github.com/chanzuckerberg/miniwdl) | Yes | Yes | SLURM | AWS Batch | | ||
| | [Terra](https://terra.bio/) | No | No | No | Azure, GCP | | ||
|
|
||
| \* Also see [WDL Runner](https://github.com/broadinstitute/wdl-runner), a script for launch WDL workflows on GCP using Cromwell | ||
|
|
||
| ### Grammars, Parsers, and Language Support | ||
|
|
||
| - The WDL [parsers repository](https://github.com/openwdl/wdl-parsers/) provides grammar definitions in various formats and generated parsers for various languages. | ||
| - [MiniWDL](https://github.com/chanzuckerberg/miniwdl) provides python bindings for WDL as well as commandline tools for validation, linting, and generating workflow input templates. | ||
| - [WOMTool](https://cromwell.readthedocs.io/en/stable/WOMtool/) is a standalone Java tool for WDL parsing, validating, linting, and diagramming. | ||
| - [wdlTools](https://github.com/dnanexus/wdlTools) - provides 1) a parser Java/Scala library, based on the [ANTLR4 grammars](https://github.com/openwdl/wdl-parsers) grammars, for WDL draft-2, 1.0, 1.1, and 2.0; and 2) command-line tools for sytanx checking, type-checking, linting, code formatting (including upgrading from older to newer WDL versions), generating documentation, and executing WDL tasks locally. | ||
| - [sprocket](https://github.com/stjude-rust-labs/sprocket) a package manager and linter for WDL files. | ||
|
|
||
| ### IDE Support | ||
|
|
||
| | IDE | Tool | | ||
| | ------------------ | ------------------------------------------------------------------------------------------------ | | ||
| | Emacs | [poly-wdl](https://github.com/jmonlong/poly-wdl) | | ||
| | Emacs | [wdl-mode](https://github.com/zhanxw/wdl-mode) | | ||
| | JetBrains | [Winstanly](https://plugins.jetbrains.com/plugin/8154-winstanley-wdl) | | ||
| | Sublime | [WDL Syntax Highlighter](https://github.com/broadinstitute/wdl-sublime-syntax-highlighter) | | ||
| | Vim | [vim-wdl](https://github.com/broadinstitute/vim-wdl) | | ||
| | Visual Studio Code | [WDL Syntax Highlighter](https://marketplace.visualstudio.com/items?itemName=broadinstitute.wdl) | | ||
|
|
||
| ### Documentation | ||
|
|
||
| - [wdl-aid](https://github.com/biowdl/wdl-aid) generates documentation for the inputs of WDL workflows, based on the parameter_meta information defined in the WDL file. | ||
| - [wdldoc](https://github.com/stjudecloud/wdldoc) | ||
|
|
||
| ### Testing | ||
|
|
||
| - [wdl-tests](https://github.com/openwdl/wdl-tests) is a collection of test cases for WDL implementations. A specification is provided for writing new tests that are compatible with automated testing frameworks. | ||
| - [Pytest-workflow](https://github.com/LUMC/pytest-workflow) is a implementation-agnostic workflow tester. It can be used with both Cromwell and MiniWDL. Uses pytest as the underlying test framework. Tests can be specified in either YAML format or python code. | ||
| - [Pytest-wdl](https://github.com/EliLillyCo/pytest-wdl) is a plugin for the pytest unit testing framework that enables testing of WDL tasks and workflows. Tests can be specified in either YAML format or python code. | ||
|
|
||
| ### Packaging | ||
|
|
||
| - [miniwdl zip](https://miniwdl.readthedocs.io/en/latest/zip.html) generates a ZIP file including a given WDL source code file and any other WDL files it imports. The ZIP file can be supplied directly to miniwdl run, which can extract it automatically. | ||
| - [wdl-packager](https://github.com/biowdl/wdl-packager) packages a workflow and all of its imports into a zip file. The zip can be used as an imports zip package for cromwell. The utility can add non-WDL files (such as the license) to the zip package and provides options to package the zip in a binary reproducible way. | ||
|
|
||
| ## Contributing | ||
|
|
||
| WDL only advances through community contributions. | ||
| While participating in discussions and submitting issues are great ways to be involved, help is also needed to implement changes to the specification. | ||
| For more information on how you can contribute, please read the [Contributing](CONTRIBUTING.md) guide. | ||
|
|
||
| ### RFC Process | ||
|
|
||
| Submitted [pull requests](https://github.com/openwdl/wdl/pulls) are subject to the [RFC Process](RFC.md). | ||
| Please review and familiarize yourself with the process if you would like to see changes submitted to the specification. | ||
|
|
||
| ## Governance | ||
|
|
||
| The WDL specification is entirely community driven; however, it is overseen by a [Governance committee](GOVERNANCE.md). | ||
| If you are interested in being involved in WDL governance, please join the [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) and post a message in the `#general` channel. | ||
| [github-main]: https://github.com/openwdl/wdl | ||
| [github-branches]: https://github.com/openwdl/wdl/branches | ||
| [github-issues]: https://github.com/openwdl/wdl/issues | ||
| [join-slack]: https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw | ||
| [wdl-docs]: https://docs.openwdl.org/en/stable |