Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI: generate workflow.json #615

Open
christad92 opened this issue Feb 28, 2024 · 6 comments
Open

CLI: generate workflow.json #615

christad92 opened this issue Feb 28, 2024 · 6 comments

Comments

@christad92
Copy link

christad92 commented Feb 28, 2024

Overview

Developers and technical users often define their own workflows to test things out locally by writing their workflows in the JSON format that is quite fiddly and requires the developer to remember the right key names and structure (which is not always intuitive).

See the Workflows section of the readme.

Background

The OpenFn platform is an open source tool for data and workflow automation which provides users with the two different types of interface: The CLI and Lightning (a web based platform that provides a GUI/Canvas for users to design their workflows, write job expressions and run work orders.)

Requirements

We should add an interactive command to the existing CLI which makes it easy to generate a workflow. The command should ask the following questions:

  • What should the workflow be called
  • What is the name of the first step
  • What is the adaptor for the next step
  • Would you like to generate another step?
  • Where would you like to output the workflow? (accept a folder, not a file, default to the working dir)

Note the following:

  • Each step should have an id auto-generated from the name. The id should be "slugified" (spaces converted to -, special characters removed etc)
  • For each step, we should generate a .js file using the step id. The workflow should point to this file (expression: "./fetch-data.js", and each expression file should contain a basic expression of the form fn((state) => state)
  • If another step is generated, the previous step should have an edge (next) generated for it, with condition !state.error
  • The user can add as many steps as they want
  • A workflow should not be given an options object

Pretty much the minimum we expect is to generate two jobs with an edge between them. Ideally you could run this in the CLI directly without any errors (this will work if the adaptor is common).

Testing

It is too hard to unit test the final automated command, but we do expect unit tests on the individual functions used by the command.

Useful Resources

@christad92 christad92 transferred this issue from OpenFn/lightning Feb 28, 2024
@christad92 christad92 added the DMP 2024 Submission for DMP label Feb 29, 2024
@christad92 christad92 changed the title CLI Workflow Tooling [DMP 2024] CLI Workflow Tooling Feb 29, 2024
@smitjiwani

This comment was marked as off-topic.

@taylordowns2000 taylordowns2000 removed the DMP 2024 Submission for DMP label Apr 8, 2024
@christad92 christad92 added the good first issue Good for newcomers label Apr 8, 2024
@glitcher007

This comment was marked as off-topic.

@alishad846

This comment was marked as off-topic.

@christad92 christad92 changed the title [DMP 2024] CLI Workflow Tooling CLI Workflow Tooling Apr 10, 2024
@taylordowns2000 taylordowns2000 removed the good first issue Good for newcomers label Apr 10, 2024
@taylordowns2000
Copy link
Member

Hi all, apologies for the confusion here. This particular issue is not being sponsored as part of DMP 2024. You can find a list of all DMP 2024 issues here: https://github.com/orgs/OpenFn/projects/3/views/3?filterQuery=is%3Aissue+label%3A%22DMP+2024%22

@ShreyanshGautam

This comment was marked as off-topic.

@josephjclark

This comment was marked as resolved.

@josephjclark josephjclark changed the title CLI Workflow Tooling CLI: generate workflow.json May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Icebox
Development

No branches or pull requests

7 participants