Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Forward reference for datasets/datalinks #116

Open
swampie opened this issue Feb 12, 2024 · 1 comment
Open

[Feature] Forward reference for datasets/datalinks #116

swampie opened this issue Feb 12, 2024 · 1 comment
Milestone

Comments

@swampie
Copy link
Member

swampie commented Feb 12, 2024

Currently seqerakit does support dataset creation but to have an e2e flow where a pipeline input can reference a Seqera Platform's dataset url is necessary to use a mix of python and yaml.

It would be great to have the possibility to define in the same yaml a forward reference for entities created at different stage of the execution:

Pseudo-code

dataset:
   name: 'input.csv'
   file: 'samplesheet/rnaseq.csv'
   reference: 'my-input'
...
pipeline:
  name: rnaseq
...
launch:
   pipeline:
     name: rnaseq
     params:
        input: $ref.my-input <- that's references the url of the previously created dataset

The behavioud could also be enforced with validation where the $ref keyword (or similar) can be parsed to check if in the timeline of the execution an entity with that name has been created.

Once available, the same can be used for data-links

@ejseqera
Copy link
Member

ejseqera commented Apr 5, 2024

I've proposed a solution for this, would you be able to kindly test it out and let me know what you think? @swampie

I am trying to avoid having to specify additional key-value pairs and use the dataset name created as a reference for retrieving the dataset URL to be used as the input value in launching, and adding a pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants