Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Add feature to emit YAMLs of existing entities #112

Draft
wants to merge 7 commits into
base: dev
Choose a base branch
from

Conversation

ejseqera
Copy link
Member

@ejseqera ejseqera commented Jan 10, 2024

Partially closes #83 to add a seqerakit dump feature to emit YAML definitions for right now only CEs and Pipelines. Also improves CLI option readability and usability.

To test:
Install this branch locally:

git checkout dump_yamls
pip install .

Run the dump command:

seqerakit dump -w <any_workspace> -p <optional_prefix>

For example:

seqerakit dump -w seqeralabs/showcase -p showcase_dump

Additionally, to see the improvements to CLI arg parser:

seqerakit -h

TODO:

  • Improve support in YAML parsing for lists
  • Potentially add support for Datasets as well
  • Refactor the DumpYaml class (kind of mess rn)
  • Add logging

@ejseqera ejseqera changed the title Add feature to emit YAMLs of existing entities Draft: Add feature to emit YAMLs of existing entities Mar 4, 2024
@ejseqera ejseqera marked this pull request as draft March 4, 2024 04:05
Copy link
Collaborator

@adamrtalbot adamrtalbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this months ago and didn't hit send


To generate definitions for objects and entities that exist within a Workspace on Seqera Platform in YAML format, you can use the `seqerakit dump` command.

To use this feature, specify the Workspace name or ID for which you want to generate YAML files for, and optionally, a prefix for your YAML filenames (this defaults to the Workspace):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just write a single file? -o output.yml?

Comment on lines +13 to +50
self.yaml_keys = {
"compute-envs": [
{
"discriminator": "type",
"name": "name",
"workspace": "workspace",
"region": "region",
"workDir": "work-dir",
"waveEnabled": "wave",
"fusion2Enabled": "fusion-v2",
"nvnmeStorageEnabled": "fast-storage",
"type": "provisioning-model",
"instanceTypes": "instance-types", # TODO
"minCpus": "min-cpus",
"maxCpus": "max-cpus",
"ebsAutoScale": "no-ebs-autoscale",
"fargateHeadEnabled": "fargate",
"gpuEnabled": "gpu",
},
], # TODO add missing keys
"pipelines": [
{
"pipeline": "url",
"description": "description",
"workspace": "workspace",
"workDir": "work-dir",
"revision": "revision",
"configProfiles": "profile", # TODO handle YAML list representation
"configText": "config",
"paramsFile": "params-file",
"resume": "resume",
"pullLatest": "pull-latest",
"stubRun": "stub-run",
"pre-run": "pre-run",
"computeEnvId": "compute-env",
},
], # TODO add support for datasets
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this work with classes that inherit this?

something like this (untested):

class ComputeEnvYaml(DumpYaml):
    def __init__(self, sp, workspace):
        DumpYaml.__init__(self, sp, workspace)
        self.yaml_keys = {
                    "discriminator": "type",
                    "name": "name",
                    "workspace": "workspace",
                    "region": "region",
                    "workDir": "work-dir",
                    "waveEnabled": "wave",
                    "fusion2Enabled": "fusion-v2",
                    "nvnmeStorageEnabled": "fast-storage",
                    "type": "provisioning-model",
                    "instanceTypes": "instance-types",  # TODO
                    "minCpus": "min-cpus",
                    "maxCpus": "max-cpus",
                    "ebsAutoScale": "no-ebs-autoscale",
                    "fargateHeadEnabled": "fargate",
                    "gpuEnabled": "gpu",
                }

It will be easier to extend the class to new objects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants