Roadmap

[jdidion]

The purpose of this discussion is to establish a roadmap for the future of WDL. First, we should agree on the release strategy. Second, we should decide on which features to include in the next version.

Current State of Affairs

• WDL 1.0 is fully supported by all known implementations
• WDL 1.1 was released in early 2021 and is fully supported in at least some implementations
• WDL 1.1.1 planned for 2023?, including:
  • Incorporation of all 1.1 errata into the spec
  • Improvement of examples to be useable as test-cases
• WDL development was begun several years ago, and was recently updated to be identical with 1.1 with the exception of:
  • The Directory data type
  • Hints
  • Elimination of several features that break backward compatibility

What's Next

On the Oct 25, 2021 governance call, the board voted to proceed with WDL 1.2 that will include Directory, Hints, and other new features, but no breaking changes. We will continue to put off 2.0 until there is greater certainty of feature stability, in order to avoid the need for 3.0 anytime soon.

Below are enumerated all the current proposals for new features that we could choose to include in WDL 1.2. I encourage people to comment on which of these features should/shouldn't be in the next release.

• Directories
  • Backport directory type to 1.2
  • Directory listings (Determining input equivalence with File checksums and Directory listings #472)
  • Secondary files (File Bundles and Secondary / Accessory Files #289)
  • Clarify that input files/directories are read-only (assumptions around modifying input files, or making new files in their directories #495)
    • This may break existing workflows depending on how the engine has already implemented this case
• Runtime/hints
  • Deprecate runtime section
  • Add new requirements section that is equivalent to the runtime section in development
  • Backport hints to 1.2
  • Inline Docker images (RFC: inline Dockerfiles for simple images #406)
  • Runtime access to runtime attributes (RFC: Runtime access to runtime attributes #418)
  • Workflow hints (RFC: workflow hints section #417)
• Other syntax changes
  • Environment variables (automatically propagate task values to environment variables #458)
  • Multi-line strings (Multi-line strings in meta, parameter_meta #221, Multi-line strings #414, Backport multi-line strings to v1.2 spec #522)
  • Make input: optional in call block, change input: to input {} (Simplify call syntax: make 'input:' keyword optional #198)
  • Enums (Support Enumerations #139)
  • Struct metadata (Parameter meta for structs #215)
    • I would add both meta and parameter_meta sections
  • Exponentiation operator (Exponentiation operator #296)
  • Namespaces
    • An additional or alternative mechanism for importing structs
    • Discussed in One idea to make imports more intuitive #214, Structs should be separed by namespaces #280, Multiple identical Struct declarations #448
• Clarification of current spec ambiguities
  • Specify behavior when specifying None as argument to non-optional call input with default (clarify how inputs with defaults are implicitly optional #464)
    • This may break existing workflows depending on how the engine has already implemented this case
  • Case-insensitive read_boolean (should read_boolean() convert inputs to lowercase before checking? #494)
  • Clarify short-circuiting of boolean operators (Clarify short-circuiting of &&, || and if/then/else expressions #199)
• Relaxation of current specification (there is a small chance of these breaking existing workflows where people are expecting these cases to fail)
  • Permissive coercion - allow coercion to Struct when there are extra keys (Clarify whether extra keys are allowed when coercing Map to Struct #482)
  • More lenient expression evaluation in interpolations (Enable all expressions to operate on optional values within interpolations #481)
• Standard library functions
  • split function (Proposal: chunk function, to create subsets of an array #398)
  • contains function (A contains style function for Array #117, Simple string "contains" checks #133)
    • This would be a polymorphic function that would work on String, Array, and Map
    • Could additionally work for Object - return true only if the object has a field with the specified name
    • Alternatively, contains_key function was proposed for Maps (map contains_key() function #305)
  • map function or list comprehensions (engine function to map over an Array #203)
  • values function (Add a function to get a list of the values from a Map #43)
  • Reading structured data from TSV and JSON (read TSV with headers #194, Reading structured data from JSON and TSV #389)
  • resolve function (resolve function #426)
  • Add optional default parameter to select_first
  • Generalize size function to operate on all collection-typed inputs (Generalize size function to operate on any collection-typed input #531)
  • Generalize length function to also operate on Map (Get length of Map #534)
  • Function for explicitly casting to a different data type (Automatic Coercions Considered Harmful #373)
• Other
  • Standard error JSON format (RFC: specification for error JSON format #419)

Voting

On the March 10, 2023 governance call, we decided to move forward with a version 1.2 specification and to commence voting on which features would be included. Voting was conducted via Google sheet.

The following features have a majority of votes (at least 5) for inclusion:

• Directories
  • Backport Directory type
  • Directory listings
  • Clarify that engines SHOULD treat input files/directories as read-only
• Runtime
  • Deprecate runtime section
  • New requirements section, which is essentially the runtime section from development
  • Backport hints section
  • Runtime access to requirements/hints
  • Workflow hints
• Other Syntax changes
  • Environment variables
  • Backport multi-line strings
  • Struct metadata
  • Make "input:" optional
  • Exponentiation operator
• Clarification of current spec ambiguities
  • Behavior when specifying None as argument to non-optional call input with default
  • Boolean short-circuiting
  • read_boolean is case-insensitive
• Relaxation of current specification
  • Allow coercion to Struct when there are extra keys
• Standard library functions
  • split (partitioning of arrays)
  • contains (string/array/map/object membership)
  • values (returns the values of a map)
  • Add optional "default" parameter to select_first
  • Generalize size function
  • Generalize length function
  • Add optional "header" parameter to read_tsv

Decisions on the following features were deferred until there is a more complete proposal:

• Secondary files
• Enums
• More lenient expression evaluation in interpolations
• Changes to read_json and read_tsv for structured data
• Explicit type casting (e.g. cast function)

[rhpvorderman]

For inclusion in WDL 1.2 I would vote for:

• Clarify whether extra keys are allowed when coercing Map to Struct #482 This is really useful.
• RFC: Runtime access to runtime attributes #418 . Runtime attributes should not influence call-caching (result reuse) but now they do because they are usually implemented in the input section to be able to set the correct tool flags. So fixing this should be high on the prioirity list.
• Namespacing for structs. In whatever form. Globally namespaced structs is a big headache. Given that this is 1.2 globally namespaced structs is still allowed. But instead of writing myStruct I should also be able to write tasks.common.structs.myStruct instead to be forward-compatible with 2.0 were global structs should be disallowed.
• RFC: workflow hints section #417 There is already a usecase for this.
• resolve function #426
• Multi-line strings in meta, parameter_meta #221 . Very useful
• Determining input equivalence with File checksums and Directory listings #472
• Parameter meta for structs #215

For all RFC's that include programming language-like features, I hope they never end up in the language. WDL stands for Workflow Description Language. I think it is best that it should remain as declarative as possible to keep the language as simple as possible. For more advanced concepts such as map, values, contains, split etc. it is possible to use a task with a command section that runs your programming language of choice that does these things. Usually things that need this are infrastructure-like things that only require 1 or 2 tasks to be executed (in an entire workflow that runs hundreds of tasks for multiple samples). That does not warrant adding extra complexity in my opinion.

[jdidion]

Thanks for your input, @rhpvorderman! It would be useful to be explicit about which items you're opposed to (and the assumption would be ambivalence about the rest).

It would be great if the rest of the governance team could provide their input as well @patmagee @mlin @geoffjentry @briandoconnor @vdauwera @cjllanwarne @illusional @vortexing

[illusional]

I think map explicitly is such a useful feature, you can already hack your way around it using scatters and I don't personally like the implicit: every function takes both a single and list, because then you're lost for any nested layers of lists (unless the spec specifically allows it).

I like the list comprehension approach, but we should decide either way, the map issue has been open for almost 4 years, so I agree it needs to be on the roadmap for a decision (+ potential implementation) to be made.

[jdidion]

Personally, I am in favor of, or at least ambivalent about, all of these.

I appreciate @rhpvorderman point about not adding programming language-like features to WDL; however there are some important considerations:

• The proposals for new standard library functions I've included are ones that have been requested by multiple users. I think we need to balance language simplicity with ease of use.
• Some of these functions would be cumbersome to implement as tasks. For example, having to write an entire task just to test if a Map contains an element or get its values.
• These functions are computationally simple, meaning there would be a relatively large amount of overhead to launch a separate task that would complete very quickly.

At the very least we should:

• Consider a way to write a task such that it can be easily optimized by the runtime, e.g. RFC: user defined functions #407 (comment)
• Provide a collection of WDL tasks that implement these common operations so users can import them rather than having to write them from scratch.

My proposal: define the WDL "extended" library. This would either be a single file with many tasks, or one file per task. The extended library can evolve separately from (and more quickly than) the spec. Because this set of tasks would be "official", runtimes could optionally provide built-in versions of the tasks for optimization purposes.

[vortexing]

Things I think look most interesting/useful/immediately helpful for novice users and should be prioritized:

• Secondary files (File Bundles and Secondary / Accessory Files File Bundles and Secondary / Accessory Files #289)
• Struct metadata (Parameter meta for structs Parameter meta for structs #215) (SUPER IMPORTANT/HELPFUL ONE!!!)
• Enums (Support Enumerations Support Enumerations #139) (Now I see how this is useful)
• split function (Proposal: split function, to create subsets of an array Proposal: chunk function, to create subsets of an array #398) (I'm of course going to say this though)
• map function or list comprehensions (engine function to map over an Array engine function to map over an Array #203) (But with the caveat that the ideal would be not using list comprehension and burden the user, but use more of an implicit map type approach)
• Reading structured data from TSV and JSON (read TSV with headers read TSV with headers #194, Reading structured data from JSON and TSV Reading structured data from JSON and TSV #389) (I really think this is a pretty big fail for WDL right now for ease of use for newbies - humans don't often like to look at jsons but they often like to run one workflow on many many different samples, but I admit I don't know how to match up the need to what needs to be in the spec). See my old issue too here: reading in Struct's and Array[Struct] from TSV/json to replace read_objects #387
• resolve function (resolve function resolve function #426)
• Standard error JSON format (RFC: specification for error JSON format RFC: specification for error JSON format #419)
• Runtime access to runtime attributes (RFC: Runtime access to runtime attributes RFC: Runtime access to runtime attributes #418) (Seems useful, I'd definitely leverage it, needs more input from folks as to how exactly to DO it though).
• Namespaces for structs (tough issue but certainly is critical/important especially for new users who are trying to leverage existing bits and bobbles).
• Permissive coercion - allow coercion to Struct when there are extra keys (Clarify whether extra keys are allowed when coercing Map to Struct Clarify whether extra keys are allowed when coercing Map to Struct #482)

Things I'm ambivalent about or are probably more important for other types of users than my realm:

• Multi-line strings (Multi-line strings in meta, parameter_meta Multi-line strings in meta, parameter_meta #221, Multi-line strings Multi-line strings #414)
• Make input: optional in call block, change input: to input {} (Simplify call syntax: make 'input:' keyword optional Simplify call syntax: make 'input:' keyword optional #198) (would love to remove commas from the input block, but probably wouldn't choose to remove the word "input" anyway due to ease of reading for newbies.
• Workflow hints (RFC: workflow hints section RFC: workflow hints section #417)
• More lenient expression evaluation in interpolations (Enable all expressions to operate on optional values within interpolations Enable all expressions to operate on optional values within interpolations #481)

Things I'm not even sure what need there was for these, not something I've personally run into, or maybe actively don't love:

• exponentiation operator (Exponentiation operator Exponentiation operator #296)
• Clarify short-circuiting of boolean operators (Clarify short-circuiting of &&, || and if/then/else expressions Clarify short-circuiting of &&, || and if/then/else expressions #199)
• contains function (A contains style function for Array A contains style function for Array #117, Simple string "contains" checks Simple string "contains" checks #133) (seems out of scope for WDL - it's not a programming language after all - what would this do for reproducibility? Maybe nothing?)
• values function (Add a function to get a list of the values from a Map Add a function to get a list of the values from a Map #43)
• Inline Docker images (RFC: inline Dockerfiles for simple images RFC: inline Dockerfiles for simple images #406)
• Environment variables (automatically propagate task values to environment variables automatically propagate task values to environment variables #458) (Feeling very resistant to this but change my mind).
• Specify behavior when specifying None as argument to non-optional call input with default (clarify how inputs with defaults are implicitly optional clarify how inputs with defaults are implicitly optional #464) (Will defer to others here)

[patmagee]

Okay this is a bit late in coming from me (I realize time totally got away from me on this one), but I think the list of features I would short-list for 1.2 would be the following:

• Any non breaking change that has already been included in the Development spec (this is a broad category and will require some time to specify)
• Directory listings (Determining input equivalence with File checksums and Directory listings #472)
• Struct metadata (Parameter meta for structs #215) I would add both meta and parameter_meta sections
• Multi-line strings (Multi-line strings in meta, parameter_meta #221, Multi-line strings #414)
• Environment variables (Support Defining Environment Variables Within a Task #504), there is now an open PR For this feature
• Workflow hints (RFC: workflow hints section #417)
• Make input: optional in call block, change input: to input {} this has actually be taken one step further in (abbreviated syntax for passthrough call inputs #365)
• Reading structured data from TSV and JSON (read TSV with headers #194, Reading structured data from JSON and TSV #389)

I am sure there are a few other things that I have missed in the list, however I am happy to start here

More generally speaking, I think we need a new process for releases. I do wonder whether the release branch model could be something that we should consider. Looking through the long list of changes that has happened since the last 1.1 release, it is confusing to figure out what has been included and what has been missed. Additionally, having the directory structure for the releases as opposed to the individuals branches has proven to be a bit challenging in my mind