[24.2] Add bwa_mem2_index directory datatype, framework enhancements for testing directories

[mvdbeek]

This is essentially a one-liner with a lot of testing work 😅.

The basic need is for a directory datatype, subclasses like bwa_mem2_index can represent most if not all reference data.
Here I've switched our primary tool testing function over to target __FETCH_DATA__ (workflow tests are already using this), and I fixed up cwl-style directory uploads which (at least currently) are not working.
It is now possible to upload directories in xml tool tests and workflow tests (via planemo and our internal framework).
Also adds format into the expression input object, so we can use format to decide if a workflow step needs to run.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[jmchilton]

Why 24.2 😅 - I mean I think switching the upload API for the tool tests is a long standing issue that I'm so excited to see resolved but it is a big thing that should probably be against dev right?

[mvdbeek]

As a nice side effect the tool framework tests are almost twice as fast, since __FETCH_DATA__ uploads are handled in celery.

[mvdbeek]

Why 24.2 😅 -

These are (almost, with the exception of being able to set metadata on composite datasets) exclusively client-side changes in tool-util that enable directory datasets to be used in tests ... that's what I meant with "essentially a one-liner". All of this should already work against galaxy going back as far as 18.05, when __FETCH_DATA__ was added. There might be minor fallout in the weekly iuc tests, but if there is I'm positive we can fix this pretty easily.

[mvdbeek]

Test failures are unrelated ... should I build a planemo branch that uses this against the IUC codebase or are we ok as is ?

[jmchilton]

exclusively client-side changes in tool-util that enable directory datasets to be used in tests

Which is an extensive code base with large implications and worthy some sort of versioning I think. I wouldn't say just because something doesn't affect the backend - it isn't part of the versioned interface of Galaxy. That said - I'm excited about this and I trust you will handle any Planemo fallout. I'm a +1 as is despite the branch target reservations.

[mvdbeek]

Thank you, I appreciate that!