Conditional execution results in collection including nulls

[pvanheus]

Describe the bug
In a workflow where some of the steps are conditionally executed and a collection (list type) is processed, the result is a collection that contains null values. For example, this workflow:

https://usegalaxy.org/u/pvanheus/w/tb-variant-analysis-v11-imported-from-url

and this history:

https://usegalaxy.org/u/pvanheus/h/smartt-samples

history item "bcftools consensus on collection 2313, collection 4778, and collection 2634: con..." tagged as "consensus_genome" contains null items for e.g. SRR26331600 which lacks sufficient mapped reads to call a consensus.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 24.1 3.dev0
(on usegalaxy.org)

Browser and Operating System
Operating System: Windows
Browser: Firefox

To Reproduce

Producing a minimal example is on my TODO list. For now:

1. Download the read from SRA project PRJNA1026351 (the list of accessions is an item in the above history)
2. Run the TB Variant Analysis v1.1 workflow
3. Check the consensus genome output

I'm opening this issue before I once again forget to do so, and will work on creating a more minimal example.

Expected behavior

Possible solutions I can imagine:

1. Create a tool to filter nulls out of a collection
2. Change the logic with regards to the outputs of tools when they are skipped due to conditional logic (i.e. don't make a null)

[mvdbeek]

1. Create a tool to filter nulls out of a collection

That's done using the pick parameter value tool, which you can find in the expression tools section in the workflow editor

2. Change the logic with regards to the outputs of tools when they are skipped due to conditional logic (i.e. don't make a null)

this is not an option, if you do this you won't be able to skip mapping over some elements of a collection (e.g. those that didn't pass QC)

Let me know if there's anything else you run into.

[pvanheus]

Just adding a is relevant here.

Also noting that for the workflow in question, the solution was to turn the status of the collection element ("TRUE" for datasets that pass the evaluation criteria and "FALSE" otherwise) into a list of element IDs that only represents that good (i.e. "TRUE") datasets. That list was then used to filter outputs, only including "good" elements in downstream analysis. For this particular use case that solution worked well. This might not always work though, because it mutates the list of elements - if you needed to combine two lists after conditional execution you wouldn't want to use this approach.