From a20434aea675ce21ea7ce3f7de70964b2821697d Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 13 Dec 2023 11:57:27 +0300 Subject: [PATCH 01/14] create cli installation page and update intro and usage --- docs/build-for-developers/cli-installation.md | 48 +++++ docs/build-for-developers/cli-intro.md | 65 +++---- docs/build-for-developers/cli-usage.md | 164 +++++++++++------- 3 files changed, 179 insertions(+), 98 deletions(-) create mode 100644 docs/build-for-developers/cli-installation.md diff --git a/docs/build-for-developers/cli-installation.md b/docs/build-for-developers/cli-installation.md new file mode 100644 index 00000000000..ac367a7dea2 --- /dev/null +++ b/docs/build-for-developers/cli-installation.md @@ -0,0 +1,48 @@ +--- +title: Using the CLI +sidebar_label: Installation and Updating +slug: /cli-installation +--- + +## **Installation** + +To install: + +```bash +npm install -g @openfn/cli +``` + +Make sure everything works by running the built-in test job: + +```bash +openfn test +``` + +Check the version: + +```bash +openfn -v +``` + +Get help: + +```bash +openfn help +``` + +## **Updating** + +You should be able to install a new version straight on top of your current +installation: + +```bash +npm install -g @openfn/cli +``` + +If this fails, try uninstalling the current version first: + +```bash +npm uninstall -g @openfn/cli +``` + +And then re-installing. diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index c4ff1120aaa..e11709474db 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -1,49 +1,40 @@ --- -title: Intro to the OpenFn CLI +title: OpenFn CLI Quick Start sidebar_label: Intro to CLI slug: /cli --- -:::info What is this tutorial? +### Introduction -- It's a _hands-on_ way to learn about the new OpenFn CLI. By following the - prompts and "challenges", a developer with a bit of Javascript experience - should be able to write, run, and debug complex, multi-step jobs with OpenFn, - using nothing but a text editor and their terminal. -- The estimated time to finish this developer challenge is 1 to 2 hours - (depending on your familiarity with the underlying concepts and tooling) -- If you are stuck and need help, please post in - [community.openfn.org](https://community.openfn.org/t/about-the-job-writing-category/11/1) +Welcome to the documentation for the @openfn/cli, an essential tool designed for +running workflows locally. It streamlines the execution, building, and testing +of OpenFn workflows for developers. -::: +### Overview -## Intro to the OpenFn CLI +This documentation serves as a comprehensive guide to the +[@openfn/cli](https://www.npmjs.com/package/@openfn/cli), replacing the +deprecated [@openfn/devtools](https://github.com/OpenFn/devtools). It introduces +new features, including: -The [@openfn/cli](https://github.com/OpenFn/kit/tree/main/packages/cli) is a -command line interface for running OpenFn workflows locally. It enables -developers to run, build, and test steps in an OpenFn workflow. +- A new runtime and compiler for executing and creating OpenFn jobs. +- Customizable logging output for enhanced visibility. +- Automatic installation of language adaptors. +- Support for the adaptors monorepo (@openfn/adaptors). -This CLI replaces [@openfn/devtools](https://github.com/OpenFn/devtools) and -provides a new suite of features and improvements, including: +### Before You Start -- a new runtime and compiler for executing and creating runnable OpenFn jobs, -- customizable logging output, -- automatic installation of language adaptors, -- and support for the adaptors monorepo - ([@openfn/adaptors](https://github.com/OpenFn/adaptors)) where all OpenFn - adaptor source code and documentation lives. +Before you begin with the @openfn/cli, make sure to follow these simple steps: -These features are designed to make it easier and more convenient for developers -to use and understand OpenFn. - -:::caution Looking for a way to execute jobs from OpenFn v1 locally? Use Core! - -If you're looking for a way to execute jobs running on the OpenFn v1 platform, -please see the documentation for **[@openfn/core](/documentation/core)** and -[Devtools](/documentation/devtools/home). - -::: - - -Learn more about CLI -[github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli) +1. **Code Editor:** Ensure you have a code editor installed on your machine. You + can use popular editors like [VS Code](https://code.visualstudio.com/) or + [Sublime](https://www.sublimetext.com/). +2. **Node.js Installation:** Install Node.js (version 18 or later): - For Linux, + Windows, or macOS, use a version manager like + [nvm](https://github.com/nvm-sh/nvm) or + [asdf](https://asdf-vm.com/guide/getting-started.html). - + [Install Node.js](https://kinsta.com/blog/how-to-install-node-js/) by + following this guide. +3. **Understand OpenFn Basics:** Have a basic understanding of OpenFn, + particularly jobs and adaptors. Check out the + [Intro section](/documentation/next) on this site. diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index a6255cca25a..d0e63949594 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -1,98 +1,140 @@ --- title: Using the CLI -sidebar_label: CLI Usage +sidebar_label: Basic Usage slug: /cli-usage --- -## Prerequisites +You're probably here to run jobs (expressions) or workflows, which the CLI makes +easy. Here's an outline of the basic usage scenarios -1. Ensure you have a code editor installed on your machine (e.g. - [VS Code](https://code.visualstudio.com/), - [Sublime](https://www.sublimetext.com/)) +### Run a job + +To run a single job, you must explicitly specify which adaptor to use.You can +find the list of publicly available [adaptors here](/adaptors). See examples +below + +- Use a shorthand (e.g., `http`): + + ```bash + openfn path/to/job.js -ia http + ``` + +- Use the full package name (e.g., `@openfn/language-http`): -2. Install NodeJs **v18 is the minimum version required** + ```bash + openfn path/to/job.js -ia @openfn/language-http + ``` - - To install a specific version of Node.js (in this case, version 18) on - Linux, Windows, or macOS, you can use a version manager such as nvm (Node - Version Manager) or any multiple runtime version manager eg: - [asdf](https://github.com/asdf-vm/asdf). These tools allow you to install - and switch between multiple versions of Node.js on the same machine. See - below for instructions for different operating systems. - - Read this article to learn how to install NodeJs in your machine - [kinsta.com/blog/how-to-install-node-js/](https://kinsta.com/blog/how-to-install-node-js/) +- Add a specific version: -3. Have a basic understanding of OpenFn—check out jobs and adaptors, at least, - in the [Intro section](documentation/next) of this site. -4. Install the OpenFn CLI with `npm install -g @openfn/cli` + ```bash + openfn path/to/job.js -ia http@2.0.0 + ``` -## CLI Usage - Key Commands +- Pass a path to a locally installed adaptor: -You’ll learn about these commands in the following challenges, but please refer -to this section for the key commands used in working with the CLI. + ```bash + openfn path/to/job.js -a http=/repo/openfn/adaptors/my-http-build + ``` -### Check the version +> Note: You MUST specify which adaptor to use. Pass the `-i` flag to +> auto-install that adaptor (it's safe to do this redundantly). + +### Compile a job + +The CLI will attempt to compile your job code into normalized Javascript. It +will do a number of things to make your code robust, portable, and easier to +debug from a pure JS perspective. ```bash -openfn version +openfn compile path/to/job.js ``` -### Get help +Will compile the openfn job and print or save the resulting js. + +### Write resulting state to disk: + +After the job finishes, the CLI writes the resulting state to disk. By default, +it creates an `output.json` next to the job file. You can specify custom paths +for the output and state files. ```bash -openfn help +openfn path/to/job.js -ia adaptor-name -o path/to/output.json -s path/to/state.json ``` -### Run a job +### Pass state through stdin and return output through stdout + +Use `-S` and `-O` to pass state through stdin and return the output through +stdout. ```bash -openfn path/to/job.js -ia {adaptor-name} +openfn path/to/job.js -ia adaptor-name -S '{"data": "Hello world!"}' -O ``` -Note: You MUST specify which adaptor to use. Pass the `-i` flag to auto-install -that adaptor (it's safe to do this redundantly). +### Adjust logging level -You can find the list of publicly available adaptors [here](/adaptors). +You can pass `-l info` or `--log info` to get more feedback about what's +happening during runtime. Below is the list of different log levels -> Path is the job to load the job from (a .js file or a dir containing a job.js -> file) For example `openfn execute hello.js ` Reads hello.js, looks for state -> and output in foo +| log level | description | +| --------------------------------------------- | -------------------------------------------------------- | +| `openfn path/to/job.js -a adaptor -l none` | Quiet mode | +| `openfn path/to/job.js -a adaptor -l default` | Top level information of what is happening | +| `openfn path/to/job.js -a adaptor -l info` | Get more feedback about runtime, cli and the job | +| `openfn path/to/job.js -a adaptor -l debug` | Get information about runtime, cli, compiler and the job | -```bash --i, --autoinstall Auto-install the language adaptor --a, --adaptors, --adaptor A language adaptor to use for the job -``` +### Maintain auto-installed adaptors repo -If an adaptor is already installed by auto install, you can use the command -without the `-i` options. i.e `openfn hello.js -a http` +- List the repo contents: -### Change log level + ```bash + openfn repo list + ``` -You can pass `-l info` or `--log info` to get more feedback about what's -happening, or `--log debug` for more details than you could ever use. Below is -the list of different log levels +- Specify the repo folder using the `OPENFN_REPO_DIR` env var: -```bash -openfn hello.js -a http -l none -``` + ```bash + export OPENFN_REPO_DIR=/path/to/repo + ``` -| log level | description | -| ------------ | -------------------------------------------------------- | -| `-l none` | Quiet mode | -| `-l default` | Top level information of what is happening | -| `-l info` | Get more feedback on what is happening openfn | -| `-l debug` | Get information about runtime, cli, compiler and the job | +- Auto-install adaptors and check if a matching version is found in the repo: -### Compilation + ```bash + openfn path/to/job.js -ia adaptor-name + ``` -The CLI will attempt to compile your job code into normalized Javascript. It -will do a number of things to make your code robust, portable, and easier to -debug from a pure JS perspective. +- Remove all adaptors from the repo: -```bash -openfn compile [path] -``` + ```bash + openfn repo clean + ``` -Will compile the openfn job and print or save the resulting js. +### Run adaptors from monorepo + +- Load from the monorepo using the `-m` flag: + + ```bash + openfn path/to/job.js -ma /path/to/monorepo + ``` -Learn more about CLI -[github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli) +- Set the monorepo location using the `OPENFN_ADAPTORS_REPO` env var: + + ```bash + export OPENFN_ADAPTORS_REPO=/path/to/monorepo + ``` + +- Include `-m` to load from the monorepo: + + ```bash + openfn path/to/job.js -ma adaptor-name + ``` + +- Remember to build an adaptor before running. + +If running a single job, you MUST specify which adaptor to use. + +## Get help + +```bash +openfn help +``` From 53f6c2d5f49a328478080cf35e5c5d3a2168cfc9 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 13 Dec 2023 11:57:54 +0300 Subject: [PATCH 02/14] add cli installation on sidebar menu --- sidebars-main.js | 1 + 1 file changed, 1 insertion(+) diff --git a/sidebars-main.js b/sidebars-main.js index c31cdabb374..29694bc44d5 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -39,6 +39,7 @@ module.exports = { label: 'Build (For Developers)', items: [ 'build-for-developers/cli-intro', + 'build-for-developers/cli-installation', 'build-for-developers/cli-usage', 'build-for-developers/cli-tutorial', 'build-for-developers/build-with-api', From 9a65316f908a273badac771b795310ba1cdd346c Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Thu, 14 Dec 2023 12:06:25 +0300 Subject: [PATCH 03/14] clean-ups and improvements --- docs/build-for-developers/cli-installation.md | 81 ++++++- docs/build-for-developers/cli-intro.md | 2 +- docs/build-for-developers/cli-tutorial.md | 211 +++++------------- docs/build-for-developers/cli-usage.md | 142 +++++++----- 4 files changed, 210 insertions(+), 226 deletions(-) diff --git a/docs/build-for-developers/cli-installation.md b/docs/build-for-developers/cli-installation.md index ac367a7dea2..9d9e65ad53c 100644 --- a/docs/build-for-developers/cli-installation.md +++ b/docs/build-for-developers/cli-installation.md @@ -4,45 +4,104 @@ sidebar_label: Installation and Updating slug: /cli-installation --- -## **Installation** +## Installation -To install: +**To install** ```bash npm install -g @openfn/cli ``` -Make sure everything works by running the built-in test job: +**Make sure everything works by running the built-in test job** ```bash openfn test ``` -Check the version: +The word `openfn` will invoke the CLI. The word `test` will invoke the test +command. + +
You should see some output like this: + + [CLI] ℹ Versions: + ▸ node.js 18.12.1 + ▸ cli 0.4.11 + ▸ runtime 0.2.2 + ▸ compiler 0.0.38 + [CLI] ℹ Running test job... + [CLI] ℹ Workflow object: + [CLI] ℹ { + "start": "start", + "jobs": [ + { + "id": "start", + "data": { + "defaultAnswer": 42 + }, + "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", + "next": { + "calculate": "!state.error" + } + }, + { + "id": "calculate", + "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", + "next": { + "result": true + } + }, + { + "id": "result", + "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" + } + ] + } + + [CLI] ✔ Compilation complete + [R/T] ♦ Starting job start + [JOB] ℹ Starting computer... + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job start in 1ms + [R/T] ♦ Starting job calculate + [JOB] ℹ Calculating to life, the universe, and everything.. + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job calculate in 1ms + [R/T] ♦ Starting job result + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job result in 0ms + [CLI] ✔ Result: 42 + +
+ +What we've just done is executed a JavaScript expression, which we call a _job_. +The output prefixed with `[JOB]` comes directly from `console.log` statements in +our job code. All other output is the CLI trying to tell us what it is doing. + +**Check the version** ```bash openfn -v ``` -Get help: +**Get help** ```bash openfn help ``` -## **Updating** +## Updating -You should be able to install a new version straight on top of your current -installation: +**You should be able to install a new version straight on top of your current +installation** ```bash npm install -g @openfn/cli ``` -If this fails, try uninstalling the current version first: +**If this fails, try uninstalling the current version first, And then +re-installing.** ```bash npm uninstall -g @openfn/cli +npm install -g @openfn/cli ``` - -And then re-installing. diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index e11709474db..23f93ca7475 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -22,7 +22,7 @@ new features, including: - Automatic installation of language adaptors. - Support for the adaptors monorepo (@openfn/adaptors). -### Before You Start +### Before you start Before you begin with the @openfn/cli, make sure to follow these simple steps: diff --git a/docs/build-for-developers/cli-tutorial.md b/docs/build-for-developers/cli-tutorial.md index 58e925fe8e6..fbe612b31ee 100644 --- a/docs/build-for-developers/cli-tutorial.md +++ b/docs/build-for-developers/cli-tutorial.md @@ -1,112 +1,13 @@ --- title: CLI Walkthrough & Challenges -sidebar_label: CLI Tutorial +sidebar_label: Walkthrough & Challenges slug: /cli-tutorial --- - ## Walkthrough & Challenges ### 1. Getting started with the CLI -Let's start by running a simple command with the CLI. Type the following into -your terminal: - -```bash -openfn test -``` - -The word `openfn` will invoke the CLI. The word `test` will invoke the test -command. - -
- You should see some output like this: - -```bash -[CLI] ℹ Versions: - ▸ node.js 18.12.1 - ▸ cli 0.0.39 - ▸ runtime 0.0.24 - ▸ compiler 0.0.32 -[CLI] ℹ Running test job... -[CLI] ℹ Workflow object: -[CLI] ℹ { - "start": "start", - "jobs": [ - { - "id": "start", - "data": { - "defaultAnswer": 42 - }, - "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", - "next": { - "calculate": "!state.error" - } - }, - { - "id": "calculate", - "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", - "next": { - "result": true - } - }, - { - "id": "result", - "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" - } - ] -} - -[CLI] ✔ Compilation complete -[R/T] ♦ Starting job start -[JOB] ℹ Starting computer... -[R/T] ℹ Operation 1 complete in 0ms -[R/T] ✔ Completed job start in 1ms -[R/T] ♦ Starting job calculate -[JOB] ℹ Calculating to life, the universe, and everything.. -[R/T] ℹ Operation 1 complete in 0ms -[R/T] ✔ Completed job calculate in 1ms -[R/T] ♦ Starting job result -[R/T] ℹ Operation 1 complete in 0ms -[R/T] ✔ Completed job result in 0ms -[CLI] ✔ Result: 42 - -``` - -
- -What we've just done is executed a JavaScript expression, which we call a _job_. -The output prefixed with `[JOB]` comes directly from `console.log` statements in -our job code. All other output is the CLI trying to tell us what it is doing. - -
-What is a job? -A job is Javascript code which follows a particular set of conventions. -Typically a job has one or more operations which perform a particular -task (like pulling information from a database, creating a record, etc.) and -return state for the next operation to use. - -The test job we just ran looks like this: - -```js -const fn = () => state => { - console.log( - 'Calculating the answer to life, the universe, and everything...' - ); - return state * 2; -}; -export default [fn()]; -``` - -You can see this (and a lot more detail) by running the test command with -debug-level logging: - -```bash -openfn test --log debug -``` - -
- #### Tasks: :::info To get started with @openfn/cli @@ -135,42 +36,49 @@ openfn test --log debug ::: -1. Create a file called `hello.js` and write the following code. +1. Create a job file called `hello.js` and write the following code. ```js console.log('Hello World!'); ``` +
+ What is a job? + A job is Javascript code which follows a particular set of conventions. + Typically a job has one or more operations which perform a particular + task (like pulling information from a database, creating a record, etc.) and + return state for the next operation to use. +
+
What is console.log? console.log is a core JavaScript language function which lets us send messages to the terminal window.
-1. Run the job using the CLI - - ```bash - openfn hello.js -o tmp/output.json - ``` +2. Run the job using the CLI -
+ ```bash + openfn hello.js -o tmp/output.json + ``` +
View expected output - ```bash - [CLI] ⚠ WARNING: No adaptor provided! - [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: - openfn job.js -a common - [CLI] ✔ Compiled from helo.js - [R/T] ♦ Starting job job-1 - [JOB] ℹ Hello World! - [R/T] ✔ Completed job job-1 in 1ms - [CLI] ✔ State written to tmp/output.json - [CLI] ✔ Finished in 17ms ✨ + ```bash + [CLI] ⚠ WARNING: No adaptor provided! + [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: + openfn job.js -a common + [CLI] ✔ Compiled from helo.js + [R/T] ♦ Starting job job-1 + [JOB] ℹ Hello World! + [R/T] ✔ Completed job job-1 in 1ms + [CLI] ✔ State written to tmp/output.json + [CLI] ✔ Finished in 17ms ✨ - ``` + ``` -
+
Note that our `console.log` statement was printed as `[JOB] Hello world!`. Using the console like this is helpful for debugging and/or understanding what's @@ -275,34 +183,34 @@ openfn getUsers.js -a http -o tmp/output.json ```
-See expected CLI logs: + See expected CLI logs: -``` -[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ -[R/T] ✔ Operation 1 complete in 581ms -[JOB] ℹ { - id: 1, - name: 'Leanne Graham', - username: 'Bret', - email: 'Sincere@april.biz', - address: { - street: 'Kulas Light', - suite: 'Apt. 556', - city: 'Gwenborough', - zipcode: '92998-3874', - geo: { lat: '-37.3159', lng: '81.1496' } - }, - phone: '1-770-736-8031 x56442', - website: 'hildegard.org', - company: { - name: 'Romaguera-Crona', - catchPhrase: 'Multi-layered client-server neural-net', - bs: 'harness real-time e-markets' + ``` + [CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ + [R/T] ✔ Operation 1 complete in 581ms + [JOB] ℹ { + id: 1, + name: 'Leanne Graham', + username: 'Bret', + email: 'Sincere@april.biz', + address: { + street: 'Kulas Light', + suite: 'Apt. 556', + city: 'Gwenborough', + zipcode: '92998-3874', + geo: { lat: '-37.3159', lng: '81.1496' } + }, + phone: '1-770-736-8031 x56442', + website: 'hildegard.org', + company: { + name: 'Romaguera-Crona', + catchPhrase: 'Multi-layered client-server neural-net', + bs: 'harness real-time e-markets' + } } -} -[R/T] ✔ Operation 2 complete in 2ms -[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ -``` + [R/T] ✔ Operation 2 complete in 2ms + [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ + ```
@@ -475,12 +383,7 @@ administrator for feedback. > What's the difference between the job you wrote and the compiled job? -2. Run a job without "strict mode" enabled. - - > What's the difference between the outputs when strict mode is enabled and - > disabled? - -3. Run a job with the log level set to `none`, and then run it again with the +2. Run a job with the log level set to `none`, and then run it again with the log level set to `debug`. > When is it appropriate to use these different log levels? @@ -1043,8 +946,10 @@ outlined below: ```json { ... - "data": "tmp/initial-data.json", + "state": { + "data": "tmp/initial-data.json", + } } ``` -::: \ No newline at end of file +::: diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index d0e63949594..80423f1bdbe 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -13,46 +13,34 @@ To run a single job, you must explicitly specify which adaptor to use.You can find the list of publicly available [adaptors here](/adaptors). See examples below -- Use a shorthand (e.g., `http`): +> Pass the `-i` flag to auto-install that adaptor (it's safe to do this +> redundantly). - ```bash - openfn path/to/job.js -ia http - ``` +**Use a shorthand (e.g., `http`)** -- Use the full package name (e.g., `@openfn/language-http`): - - ```bash - openfn path/to/job.js -ia @openfn/language-http - ``` - -- Add a specific version: - - ```bash - openfn path/to/job.js -ia http@2.0.0 - ``` +```bash +openfn path/to/job.js -ia http +``` -- Pass a path to a locally installed adaptor: +**Use the full package name (e.g., `@openfn/language-http`)** - ```bash - openfn path/to/job.js -a http=/repo/openfn/adaptors/my-http-build - ``` +```bash +openfn path/to/job.js -ia @openfn/language-http +``` -> Note: You MUST specify which adaptor to use. Pass the `-i` flag to -> auto-install that adaptor (it's safe to do this redundantly). +**Add a specific version** -### Compile a job +```bash +openfn path/to/job.js -ia http@2.0.0 +``` -The CLI will attempt to compile your job code into normalized Javascript. It -will do a number of things to make your code robust, portable, and easier to -debug from a pure JS perspective. +**Pass a path to a locally installed adaptor** ```bash -openfn compile path/to/job.js +openfn path/to/job.js -a http=/repo/openfn/adaptors/my-http-build ``` -Will compile the openfn job and print or save the resulting js. - -### Write resulting state to disk: +### Write resulting state to disk After the job finishes, the CLI writes the resulting state to disk. By default, it creates an `output.json` next to the job file. You can specify custom paths @@ -85,56 +73,88 @@ happening during runtime. Below is the list of different log levels ### Maintain auto-installed adaptors repo -- List the repo contents: +**List the repo contents** - ```bash - openfn repo list - ``` +```bash +openfn repo list +``` -- Specify the repo folder using the `OPENFN_REPO_DIR` env var: +**Specify the repo folder using the `OPENFN_REPO_DIR` env var** - ```bash - export OPENFN_REPO_DIR=/path/to/repo - ``` +```bash +export OPENFN_REPO_DIR=/path/to/repo +``` -- Auto-install adaptors and check if a matching version is found in the repo: +**Auto-install adaptors and check if a matching version is found in the repo** - ```bash - openfn path/to/job.js -ia adaptor-name - ``` +```bash +openfn path/to/job.js -ia adaptor-name +``` -- Remove all adaptors from the repo: +**Remove all adaptors from the repo** - ```bash - openfn repo clean - ``` +```bash +openfn repo clean +``` ### Run adaptors from monorepo -- Load from the monorepo using the `-m` flag: +**Load from the monorepo using the `-m` flag** - ```bash - openfn path/to/job.js -ma /path/to/monorepo - ``` +```bash +openfn path/to/job.js -ma /path/to/monorepo +``` -- Set the monorepo location using the `OPENFN_ADAPTORS_REPO` env var: +**Set the monorepo location using the `OPENFN_ADAPTORS_REPO` env var** - ```bash - export OPENFN_ADAPTORS_REPO=/path/to/monorepo - ``` +```bash +export OPENFN_ADAPTORS_REPO=/path/to/monorepo +``` -- Include `-m` to load from the monorepo: +**Include `-m` to load from the monorepo** - ```bash - openfn path/to/job.js -ma adaptor-name - ``` +```bash +openfn path/to/job.js -ma adaptor-name +``` -- Remember to build an adaptor before running. +> Remember to build an adaptor before running. + +### Run a workflow + +
+ A workflow has a structure like this + +```json +{ + "start": "a", // optionally specify the start node (defaults to jobs[0]) + "jobs": [ + { + "id": "a", + "expression": "fn((state) => state)", // code or a path + "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional) + "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state) + "configuration": {}, // Use this to pass credentials + "next": { + // This object defines which jobs to call next + // All edges returning true will run + // If there are no next edges, the workflow will end + "b": true, + "c": { + "condition": "!state.error" // Not that this is an expression, not a function + } + } + } + ] +} +``` -If running a single job, you MUST specify which adaptor to use. +
-## Get help +To run a workflow ```bash -openfn help +openfn path/to/workflow.json -i ``` + +Here is the detailed tutorial about +[running workflows](cli-tutorial#8-running-workflows) From 251c8aee145b6ae1cb3429e6eaf2e005b098c7ff Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Thu, 14 Dec 2023 12:07:07 +0300 Subject: [PATCH 04/14] remove strict-mode --- versioned_docs/version-legacy/cli.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/versioned_docs/version-legacy/cli.md b/versioned_docs/version-legacy/cli.md index 270cdfef440..8ad7ee59874 100644 --- a/versioned_docs/version-legacy/cli.md +++ b/versioned_docs/version-legacy/cli.md @@ -535,12 +535,7 @@ administrator for feedback. > What's the difference between the job you wrote and the compiled job? -2. Run a job without "strict mode" enabled. - - > What's the difference between the outputs when strict mode is enabled and - > disabled? - -3. Run a job with the log level set to `none`, and then run it again with the +2. Run a job with the log level set to `none`, and then run it again with the log level set to `debug`. > When is it appropriate to use these different log levels? From ee04758f320bdef5206174951057dd5b34096b72 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Thu, 14 Dec 2023 15:26:46 +0300 Subject: [PATCH 05/14] improvements --- docs/build-for-developers/cli-installation.md | 5 +---- docs/build-for-developers/cli-tutorial.md | 2 +- docs/build-for-developers/cli-usage.md | 9 ++++----- 3 files changed, 6 insertions(+), 10 deletions(-) diff --git a/docs/build-for-developers/cli-installation.md b/docs/build-for-developers/cli-installation.md index 9d9e65ad53c..341820b9ae1 100644 --- a/docs/build-for-developers/cli-installation.md +++ b/docs/build-for-developers/cli-installation.md @@ -72,10 +72,7 @@ command. [CLI] ✔ Result: 42 - -What we've just done is executed a JavaScript expression, which we call a _job_. -The output prefixed with `[JOB]` comes directly from `console.log` statements in -our job code. All other output is the CLI trying to tell us what it is doing. +All other output is the CLI telling us what it is doing internally **Check the version** diff --git a/docs/build-for-developers/cli-tutorial.md b/docs/build-for-developers/cli-tutorial.md index fbe612b31ee..920c204db41 100644 --- a/docs/build-for-developers/cli-tutorial.md +++ b/docs/build-for-developers/cli-tutorial.md @@ -69,7 +69,7 @@ slug: /cli-tutorial [CLI] ⚠ WARNING: No adaptor provided! [CLI] ⚠ This job will probably fail. Pass an adaptor with the -a flag, eg: openfn job.js -a common - [CLI] ✔ Compiled from helo.js + [CLI] ✔ Compiled from hello.js [R/T] ♦ Starting job job-1 [JOB] ℹ Hello World! [R/T] ✔ Completed job job-1 in 1ms diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index 80423f1bdbe..c612616a6c9 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -50,13 +50,12 @@ for the output and state files. openfn path/to/job.js -ia adaptor-name -o path/to/output.json -s path/to/state.json ``` -### Pass state through stdin and return output through stdout +### Return resulting state through stdout -Use `-S` and `-O` to pass state through stdin and return the output through -stdout. +Use `-O` to return the output through stdout. ```bash -openfn path/to/job.js -ia adaptor-name -S '{"data": "Hello world!"}' -O +openfn path/to/job.js -ia adaptor-name -O ``` ### Adjust logging level @@ -140,7 +139,7 @@ openfn path/to/job.js -ma adaptor-name // If there are no next edges, the workflow will end "b": true, "c": { - "condition": "!state.error" // Not that this is an expression, not a function + "condition": "!state.error" // Note that this is an expression, not a function } } } From 7bdae2869eb7e7455306c268a33409320300e2ea Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Thu, 14 Dec 2023 15:27:34 +0300 Subject: [PATCH 06/14] remove adaptors monorepo section --- docs/build-for-developers/cli-usage.md | 22 ---------------------- 1 file changed, 22 deletions(-) diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index c612616a6c9..243d72b3171 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -96,28 +96,6 @@ openfn path/to/job.js -ia adaptor-name openfn repo clean ``` -### Run adaptors from monorepo - -**Load from the monorepo using the `-m` flag** - -```bash -openfn path/to/job.js -ma /path/to/monorepo -``` - -**Set the monorepo location using the `OPENFN_ADAPTORS_REPO` env var** - -```bash -export OPENFN_ADAPTORS_REPO=/path/to/monorepo -``` - -**Include `-m` to load from the monorepo** - -```bash -openfn path/to/job.js -ma adaptor-name -``` - -> Remember to build an adaptor before running. - ### Run a workflow
From 253182fa9d194b7b0f8108ed43082acb143118ee Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 18 Dec 2023 12:09:07 +0300 Subject: [PATCH 07/14] combine intro and installation Into get started page --- docs/build-for-developers/cli-installation.md | 104 -------------- docs/build-for-developers/cli-intro.md | 132 +++++++++++++++--- 2 files changed, 115 insertions(+), 121 deletions(-) delete mode 100644 docs/build-for-developers/cli-installation.md diff --git a/docs/build-for-developers/cli-installation.md b/docs/build-for-developers/cli-installation.md deleted file mode 100644 index 341820b9ae1..00000000000 --- a/docs/build-for-developers/cli-installation.md +++ /dev/null @@ -1,104 +0,0 @@ ---- -title: Using the CLI -sidebar_label: Installation and Updating -slug: /cli-installation ---- - -## Installation - -**To install** - -```bash -npm install -g @openfn/cli -``` - -**Make sure everything works by running the built-in test job** - -```bash -openfn test -``` - -The word `openfn` will invoke the CLI. The word `test` will invoke the test -command. - -
You should see some output like this: - - [CLI] ℹ Versions: - ▸ node.js 18.12.1 - ▸ cli 0.4.11 - ▸ runtime 0.2.2 - ▸ compiler 0.0.38 - [CLI] ℹ Running test job... - [CLI] ℹ Workflow object: - [CLI] ℹ { - "start": "start", - "jobs": [ - { - "id": "start", - "data": { - "defaultAnswer": 42 - }, - "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", - "next": { - "calculate": "!state.error" - } - }, - { - "id": "calculate", - "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", - "next": { - "result": true - } - }, - { - "id": "result", - "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" - } - ] - } - - [CLI] ✔ Compilation complete - [R/T] ♦ Starting job start - [JOB] ℹ Starting computer... - [R/T] ℹ Operation 1 complete in 0ms - [R/T] ✔ Completed job start in 1ms - [R/T] ♦ Starting job calculate - [JOB] ℹ Calculating to life, the universe, and everything.. - [R/T] ℹ Operation 1 complete in 0ms - [R/T] ✔ Completed job calculate in 1ms - [R/T] ♦ Starting job result - [R/T] ℹ Operation 1 complete in 0ms - [R/T] ✔ Completed job result in 0ms - [CLI] ✔ Result: 42 - -
-All other output is the CLI telling us what it is doing internally - -**Check the version** - -```bash -openfn -v -``` - -**Get help** - -```bash -openfn help -``` - -## Updating - -**You should be able to install a new version straight on top of your current -installation** - -```bash -npm install -g @openfn/cli -``` - -**If this fails, try uninstalling the current version first, And then -re-installing.** - -```bash -npm uninstall -g @openfn/cli -npm install -g @openfn/cli -``` diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 23f93ca7475..1a410ba4e08 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -1,26 +1,20 @@ --- -title: OpenFn CLI Quick Start -sidebar_label: Intro to CLI +title: Get started with the OpenFn CLI +sidebar_label: Get started slug: /cli --- -### Introduction +#### Build and test your automated workflows and integrations via the command line. -Welcome to the documentation for the @openfn/cli, an essential tool designed for -running workflows locally. It streamlines the execution, building, and testing -of OpenFn workflows for developers. +The OpenFn CLI is a developer tool to help you build, test, and manage your +workflows and integration with OpenFn directly from the command line. It’s +simple to install, works on macOS, Windows, and Linux, and offers a range of +functionality to enhance your developer experience with OpenFn. You can use the +OpenFn CLI to: -### Overview - -This documentation serves as a comprehensive guide to the -[@openfn/cli](https://www.npmjs.com/package/@openfn/cli), replacing the -deprecated [@openfn/devtools](https://github.com/OpenFn/devtools). It introduces -new features, including: - -- A new runtime and compiler for executing and creating OpenFn jobs. -- Customizable logging output for enhanced visibility. -- Automatic installation of language adaptors. -- Support for the adaptors monorepo (@openfn/adaptors). +- Securely run OpenFn jobs and workflows +- Troubleshoot and debug OpenFn jobs +- Deployment of workflows to OpenFn ### Before you start @@ -38,3 +32,107 @@ Before you begin with the @openfn/cli, make sure to follow these simple steps: 3. **Understand OpenFn Basics:** Have a basic understanding of OpenFn, particularly jobs and adaptors. Check out the [Intro section](/documentation/next) on this site. + +### Install the OpenFn CLI + +To download the latest version of +[@openfn/cli](https://www.npmjs.com/package/@openfn/cli), on the command line, +run the following command. + +```bash +npm install -g @openfn/cli +``` + +**Make sure everything works by running the built-in test job** + +```bash +openfn test +``` + +The word `openfn` will invoke the CLI. The word `test` will invoke the test +command. + +
Expand to see the expected output. + + [CLI] ℹ Versions: + ▸ node.js 18.12.1 + ▸ cli 0.4.11 + ▸ runtime 0.2.2 + ▸ compiler 0.0.38 + [CLI] ℹ Running test job... + [CLI] ℹ Workflow object: + [CLI] ℹ { + "start": "start", + "jobs": [ + { + "id": "start", + "data": { + "defaultAnswer": 42 + }, + "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()", + "next": { + "calculate": "!state.error" + } + }, + { + "id": "calculate", + "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()", + "next": { + "result": true + } + }, + { + "id": "result", + "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()" + } + ] + } + + [CLI] ✔ Compilation complete + [R/T] ♦ Starting job start + [JOB] ℹ Starting computer... + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job start in 1ms + [R/T] ♦ Starting job calculate + [JOB] ℹ Calculating to life, the universe, and everything.. + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job calculate in 1ms + [R/T] ♦ Starting job result + [R/T] ℹ Operation 1 complete in 0ms + [R/T] ✔ Completed job result in 0ms + [CLI] ✔ Result: 42 + +
+ +All other output is the CLI telling us what it is doing internally. + +**Check the version** + +```bash +openfn -v +``` + +**Get help** + +```bash +openfn help +``` + +### Update the OpenFn CLI + +To install a new version straight on top of your current installation, run the +following command. + +```bash +npm install -g @openfn/cli +``` + +### Troubleshoot Installation + +If you encounter installation issues, try uninstalling the current version +first, And then re-installing. + +```bash +npm uninstall -g @openfn/cli +npm install -g @openfn/cli +``` From 8f6f932b5f03da6dd6f0eb7f26cd6bd2ec782fe4 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 18 Dec 2023 12:09:42 +0300 Subject: [PATCH 08/14] remove installation page --- sidebars-main.js | 1 - 1 file changed, 1 deletion(-) diff --git a/sidebars-main.js b/sidebars-main.js index 29694bc44d5..c31cdabb374 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -39,7 +39,6 @@ module.exports = { label: 'Build (For Developers)', items: [ 'build-for-developers/cli-intro', - 'build-for-developers/cli-installation', 'build-for-developers/cli-usage', 'build-for-developers/cli-tutorial', 'build-for-developers/build-with-api', From 64e1c51e303f0d94bbdfbfbb8d5725f26133f14b Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 18 Dec 2023 17:33:04 +0300 Subject: [PATCH 09/14] seperate docs walkthrough and challange --- docs/build-for-developers/cli-challenges.md | 154 ++++++++ docs/build-for-developers/cli-intro.md | 12 +- docs/build-for-developers/cli-usage.md | 49 ++- .../{cli-tutorial.md => cli-walkthrough.md} | 363 +++++++----------- 4 files changed, 324 insertions(+), 254 deletions(-) create mode 100644 docs/build-for-developers/cli-challenges.md rename docs/build-for-developers/{cli-tutorial.md => cli-walkthrough.md} (72%) diff --git a/docs/build-for-developers/cli-challenges.md b/docs/build-for-developers/cli-challenges.md new file mode 100644 index 00000000000..377eea728b5 --- /dev/null +++ b/docs/build-for-developers/cli-challenges.md @@ -0,0 +1,154 @@ +--- +title: CLI Challenges +sidebar_label: CLI Challenges +slug: /cli-challenges +--- + +#### 🏆 Challenge: Write a job that prints your name + +1. Modify `hello.js` to print your name. +2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. +3. Validate that you receive the logs below: + +```bash +[CLI] ✔ Compiled job from hello.js +[JOB] ℹ My name is { YourName } +[R/T] ✔ Operation 1 complete in 0ms +[CLI] ✔ Writing output to tmp/output.json +[CLI] ✔ Done in 366ms! ✨ +``` + +--- + +#### 🏆 Challenge: Get and inspect data via HTTP + +Using the +[https://jsonplaceholder.typicode.com/users](https://jsonplaceholder.typicode.com/users) +API, get a list of users and print the first user object. + +1. Create file called `getUsers.js` and write your operation to fetch the user. +2. Run the job using the OpenFn/cli + `openfn getUsers.js -a http -o tmp/output.json`. +3. Validate that you receive this expected CLI logs: + +```bash +openfn getUsers.js -a http -o tmp/output.json +``` + +
+ Expand to see expected CLI logs + +``` +[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ +[R/T] ✔ Operation 1 complete in 581ms +[JOB] ℹ { + id: 1, + name: 'Leanne Graham', + username: 'Bret', + email: 'Sincere@april.biz', + address: { + street: 'Kulas Light', + suite: 'Apt. 556', + city: 'Gwenborough', + zipcode: '92998-3874', + geo: { lat: '-37.3159', lng: '81.1496' } + }, + phone: '1-770-736-8031 x56442', + website: 'hildegard.org', + company: { + name: 'Romaguera-Crona', + catchPhrase: 'Multi-layered client-server neural-net', + bs: 'harness real-time e-markets' + } +} +[R/T] ✔ Operation 2 complete in 2ms +[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ +``` + +
+ +:::warning Placeholder Data + +The data displayed in this CLI logs is generated from a +[JSONPlaceholder](https://jsonplaceholder.typicode.com/) API and does not +represent real-world information. It is intended for testing and development +purposes only. + +For accurate testing, consider using real data from your API or service. + +::: + +--- + +#### 🏆 Challenge: Fetch Covid-19 metadata + +1. Using the [disease.sh API](https://disease.sh/), write an operation that + returns all covid-19 metadata. + +:::tip + +`https://disease.sh/v3/covid-19/` as your **baseUrl** in `state.configuration` + +::: + +2. Validate your output: there are a lot of ways you might choose to format or + display this data. Share your results with your administrator for feedback. + +--- + +#### 🏆 Challenge: Practice CLI arguments and commands + +Perform these tasks and submit answers to the discussion questions to your +administrator for feedback. + +1. Compile a openfn job (**hello.js**). + + > What's the difference between the job you wrote and the compiled job? + +2. Run a job with the log level set to `none`, and then run it again with the + log level set to `debug`. + + > When is it appropriate to use these different log levels? + +--- + +#### 🏆 Challenge: extract names & emails + +Using +[https://jsonplaceholder.typicode.com/posts/1/comments](https://jsonplaceholder.typicode.com/posts/1/comments) +API fetch comments for post with id 1 and extract name and email from each +comment in that post + +1. Get post all comments for post id 1 +2. Extract name and email from comments +3. Log the extracted data from comments + +Discuss the results with your administrator. + +--- + +#### 🏆 Challenge: control error messages + +Debug what is causing an error on the following line of code and display the +error message + +```jsx +// Get post where id is 180 +get('posts/180'); +``` + +Discuss the results with your administrator. + +--- + +#### 🏆 Challenge: Reduce, filter, and map + +Using Javascript globals i.e `Array.reduce`, `Array.filter` or `Array.map`, +build function that will get posts by user id. + +1. Create a file called job1.js +2. Add the 1st operation which is get all posts +3. Add 2nd operation which has a function that filter posts by id +4. Use the function from 2nd operation to get all post for user id 1 + +Discuss the results with your administrator. diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md index 1a410ba4e08..1f1583439d3 100644 --- a/docs/build-for-developers/cli-intro.md +++ b/docs/build-for-developers/cli-intro.md @@ -16,6 +16,8 @@ OpenFn CLI to: - Troubleshoot and debug OpenFn jobs - Deployment of workflows to OpenFn +--- + ### Before you start Before you begin with the @openfn/cli, make sure to follow these simple steps: @@ -33,6 +35,8 @@ Before you begin with the @openfn/cli, make sure to follow these simple steps: particularly jobs and adaptors. Check out the [Intro section](/documentation/next) on this site. +--- + ### Install the OpenFn CLI To download the latest version of @@ -118,6 +122,8 @@ openfn -v openfn help ``` +--- + ### Update the OpenFn CLI To install a new version straight on top of your current installation, run the @@ -127,10 +133,12 @@ following command. npm install -g @openfn/cli ``` +--- + ### Troubleshoot Installation -If you encounter installation issues, try uninstalling the current version -first, And then re-installing. +If you encounter installation issues, try uninstalling the current version first +and then re-installing. ```bash npm uninstall -g @openfn/cli diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index 243d72b3171..7e88d41bc0b 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -1,50 +1,57 @@ --- -title: Using the CLI +title: Basic usage of OpenFn CLI sidebar_label: Basic Usage slug: /cli-usage --- +#### Execute a job, run a workflow, adjust logging, maintain adaptors, and save the state. + You're probably here to run jobs (expressions) or workflows, which the CLI makes -easy. Here's an outline of the basic usage scenarios +easy. Keep reading for an outline of the basic usage scenarios of the CLI. + +--- ### Run a job -To run a single job, you must explicitly specify which adaptor to use.You can +To run a single job, you must explicitly specify which adaptor to use. You can find the list of publicly available [adaptors here](/adaptors). See examples -below +below. > Pass the `-i` flag to auto-install that adaptor (it's safe to do this > redundantly). -**Use a shorthand (e.g., `http`)** +**Use a shorthand (e.g., `http`):** ```bash openfn path/to/job.js -ia http ``` -**Use the full package name (e.g., `@openfn/language-http`)** +**Use the full package name (e.g., `@openfn/language-http`):** ```bash openfn path/to/job.js -ia @openfn/language-http ``` -**Add a specific version** +**Add a specific version:** ```bash openfn path/to/job.js -ia http@2.0.0 ``` -**Pass a path to a locally installed adaptor** +**Pass a path to a locally installed adaptor:** ```bash openfn path/to/job.js -a http=/repo/openfn/adaptors/my-http-build ``` +--- + ### Write resulting state to disk After the job finishes, the CLI writes the resulting state to disk. By default, -it creates an `output.json` next to the job file. You can specify custom paths -for the output and state files. +it creates an `output.json` next to the job file. + +**You can specify custom paths for the output and state files:** ```bash openfn path/to/job.js -ia adaptor-name -o path/to/output.json -s path/to/state.json @@ -52,12 +59,14 @@ openfn path/to/job.js -ia adaptor-name -o path/to/output.json -s path/to/state.j ### Return resulting state through stdout -Use `-O` to return the output through stdout. +**Use `-O` to return the output through stdout:** ```bash openfn path/to/job.js -ia adaptor-name -O ``` +--- + ### Adjust logging level You can pass `-l info` or `--log info` to get more feedback about what's @@ -70,32 +79,36 @@ happening during runtime. Below is the list of different log levels | `openfn path/to/job.js -a adaptor -l info` | Get more feedback about runtime, cli and the job | | `openfn path/to/job.js -a adaptor -l debug` | Get information about runtime, cli, compiler and the job | +--- + ### Maintain auto-installed adaptors repo -**List the repo contents** +**List the repo contents:** ```bash openfn repo list ``` -**Specify the repo folder using the `OPENFN_REPO_DIR` env var** +**Specify the repo folder using the `OPENFN_REPO_DIR` env var:** ```bash export OPENFN_REPO_DIR=/path/to/repo ``` -**Auto-install adaptors and check if a matching version is found in the repo** +**Auto-install adaptors and check if a matching version is found in the repo:** ```bash openfn path/to/job.js -ia adaptor-name ``` -**Remove all adaptors from the repo** +**Remove all adaptors from the repo:** ```bash openfn repo clean ``` +--- + ### Run a workflow
@@ -127,11 +140,11 @@ openfn repo clean
-To run a workflow +**To run a workflow:** ```bash openfn path/to/workflow.json -i ``` -Here is the detailed tutorial about -[running workflows](cli-tutorial#8-running-workflows) +Check out this detailed [tutorial](cli-tutorial#8-running-workflows) on running +workflows via the CLI. diff --git a/docs/build-for-developers/cli-tutorial.md b/docs/build-for-developers/cli-walkthrough.md similarity index 72% rename from docs/build-for-developers/cli-tutorial.md rename to docs/build-for-developers/cli-walkthrough.md index 920c204db41..23bf275c0b8 100644 --- a/docs/build-for-developers/cli-tutorial.md +++ b/docs/build-for-developers/cli-walkthrough.md @@ -1,15 +1,11 @@ --- -title: CLI Walkthrough & Challenges -sidebar_label: Walkthrough & Challenges -slug: /cli-tutorial +title: CLI Walkthrough +sidebar_label: CLI Walkthrough +slug: /cli-walkthrough --- -## Walkthrough & Challenges - ### 1. Getting started with the CLI -#### Tasks: - :::info To get started with @openfn/cli 1. Create a new folder for the repository you'll be working on by running the @@ -84,20 +80,6 @@ Note that our `console.log` statement was printed as `[JOB] Hello world!`. Using the console like this is helpful for debugging and/or understanding what's happening inside our jobs. -#### 🏆 Challenge: Write a job that prints your name - -1. Modify `hello.js` to print your name. -2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. -3. Validate that you receive the logs below: - -```bash -[CLI] ✔ Compiled job from hello.js -[JOB] ℹ My name is { YourName } -[R/T] ✔ Operation 1 complete in 0ms -[CLI] ✔ Writing output to tmp/output.json -[CLI] ✔ Done in 366ms! ✨ -``` - ### 2. Using adaptor helper functions Adaptors are Javascript or Typescript modules that provide OpenFn users with a @@ -141,7 +123,7 @@ Since it is our first time using the `http` adaptor, we are installing the adaptor using `-i` argument
- 3. See expected CLI logs + 3. Expand to see expected CLI logs ```bash [CLI] ✔ Installing packages... @@ -167,53 +149,6 @@ adaptor using `-i` argument
-#### 🏆 Challenge: Get and inspect data via HTTP - -Using the -[https://jsonplaceholder.typicode.com/users](https://jsonplaceholder.typicode.com/users) -API, get a list of users and print the first user object. - -1. Create file called `getUsers.js` and write your operation to fetch the user. -2. Run the job using the OpenFn/cli - `openfn getUsers.js -a http -o tmp/output.json`. -3. Validate that you receive this expected CLI logs: - -```bash -openfn getUsers.js -a http -o tmp/output.json -``` - -
- See expected CLI logs: - - ``` - [CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ - [R/T] ✔ Operation 1 complete in 581ms - [JOB] ℹ { - id: 1, - name: 'Leanne Graham', - username: 'Bret', - email: 'Sincere@april.biz', - address: { - street: 'Kulas Light', - suite: 'Apt. 556', - city: 'Gwenborough', - zipcode: '92998-3874', - geo: { lat: '-37.3159', lng: '81.1496' } - }, - phone: '1-770-736-8031 x56442', - website: 'hildegard.org', - company: { - name: 'Romaguera-Crona', - catchPhrase: 'Multi-layered client-server neural-net', - bs: 'harness real-time e-markets' - } - } - [R/T] ✔ Operation 2 complete in 2ms - [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ - ``` - -
- ### 3. Understanding `state` If a job expression is a set of instructions for a chef (a recipe?) then the @@ -222,7 +157,8 @@ bundle. See ["It all starts with state​"](/articles/2021/07/05/wrapping-my-head-around-jobs/#it-all-starts-with-state) in the knowledge base for extra context. -It usually looks something like this +
+ It usually looks something like this ```json { @@ -242,6 +178,8 @@ It usually looks something like this } ``` +
+ #### `state.configuration` This key is where we put credentials which are used to authorize connections to @@ -278,7 +216,8 @@ Specify a path to your `state.json` file with this command: openfn hello.js -a http -s tmp/state.json -o tmp/output.json ``` -Expected CLI logs +
+ Expand to see expected CLI logs ``` [CLI] ✔ Compiled job from hello.js @@ -289,6 +228,8 @@ GET request succeeded with 200 ✓ [CLI] ✔ Done in 1.222s! ✨ ``` +
+ #### How can we use state? Each adaptor has a configuration schema that's recommended for use in your @@ -307,20 +248,28 @@ of how to set up `state.configuration` for `language-http`. 1. Update your `state.json` to look like this: - ```json title=state.json - { - "configuration": { - "baseUrl": "https://jsonplaceholder.typicode.com" - } - } - ``` +
+ Expand to see state.json - Since we have update our configuration in our `state.json` we can now use - `get()` helper function without the need to specify the **baseUrl**—i.e - `get('posts')` + ```json title=state.json + { + "configuration": { + "baseUrl": "https://jsonplaceholder.typicode.com" + } + } + ``` + +
+ +Since we have update our configuration in our `state.json` we can now use +`get()` helper function without the need to specify the **baseUrl**—i.e +`get('posts')` 2. Update your `getPosts.js` job to look like this: +
+ Expand to see getPosts.js + ```js title="getPosts.js" // Get all posts get('posts'); @@ -332,63 +281,39 @@ of how to set up `state.configuration` for `language-http`. }); ``` +
+ 3. Now run the job using the following command ```bash openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json ``` - And validate that you see the expected CLI logs: +
+ And validate that you see the expected CLI logs: ```bash [CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ Operation 1 complete in 120ms [JOB] ℹ { - userId: 1, - id: 1, - title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', - body: 'quia et suscipit\n' + - 'suscipit recusandae consequuntur expedita et cum\n' + - 'reprehenderit molestiae ut ut quas totam\n' + - 'nostrum rerum est autem sunt rem eveniet architecto' + userId: 1, + id: 1, + title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit', + body: 'quia et suscipit\n' + + 'suscipit recusandae consequuntur expedita et cum\n' + + 'reprehenderit molestiae ut ut quas totam\n' + + 'nostrum rerum est autem sunt rem eveniet architecto' } [R/T] ✔ Operation 2 complete in 0ms [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 470ms! ✨ - ``` -#### 🏆 Challenge: Fetch Covid-19 metadata - -1. Using the [disease.sh API](https://disease.sh/), write an operation that - returns all covid-19 metadata. - -:::tip - -`https://disease.sh/v3/covid-19/` as your **baseUrl** in `state.configuration` - -::: - -2. Validate your output: there are a lot of ways you might choose to format or - display this data. Share your results with your administrator for feedback. - -### 4. Additional arguments and commands - -#### 🏆 Challenge: Practice CLI arguments and commands - -Perform these tasks and submit answers to the discussion questions to your -administrator for feedback. - -1. Compile a openfn job (**hello.js**). - - > What's the difference between the job you wrote and the compiled job? - -2. Run a job with the log level set to `none`, and then run it again with the - log level set to `debug`. + ``` - > When is it appropriate to use these different log levels? +
-### 5. Manipulating data in a sequence of operations +### 4. Transform data in a sequence of operations In most cases you need to manipulate, clean, or transform data at some step in your workflow. For example after we get data from the @@ -397,36 +322,31 @@ by user id. The example below shows how we can: 1. get all posts and return them in `state.data` 2. group returned posts by `userId` -3. log posts with userId 1 +3. log posts with userId `1` -##### Example: +
+Expand to see example: ```js title="getPosts.js" // Get all posts get('posts'); -// Group posts by user id -fn(state => { - const posts = state.data; +// Group posts by user id fn(state => { const posts = state.data; - // Group posts by userId - const groupPostsByUserId = posts.reduce((acc, post) => { - const existingValue = acc[post.userId] || []; - return { ...acc, [post.userId]: [...existingValue, post] }; - }, {}); +// Group posts by userId const groupPostsByUserId = posts.reduce((acc, post) => +{ const existingValue = acc[post.userId] || []; return { ...acc, [post.userId]: +[...existingValue, post] }; }, {}); - // console.log(groupPostsByUserId); - return { ...state, groupPostsByUserId }; -}); +// console.log(groupPostsByUserId); return { ...state, groupPostsByUserId }; }); -// Log posts where userId = 1 -fn(state => { - const { groupPostsByUserId } = state; - console.log('Post with userId 1', groupPostsByUserId[1]); - return state; +// Log posts where userId = 1 fn(state => { const { groupPostsByUserId } = +state; console.log('Post with userId 1', groupPostsByUserId[1]); return state; }); + ``` +
+
What is array.reduce? The reduce() method applies a function against an accumulator and @@ -438,17 +358,14 @@ the sum of all the elements in an array: ##### JavaScript Demo: `Array.reduce()` ``` + const array1 = [1, 2, 3, 4]; -// 0 + 1 + 2 + 3 + 4 -const initialValue = 0; -const sumWithInitial = array1.reduce( - (accumulator, currentValue) => accumulator + currentValue, - initialValue -); +// 0 + 1 + 2 + 3 + 4 const initialValue = 0; const sumWithInitial = +array1.reduce( (accumulator, currentValue) => accumulator + currentValue, +initialValue ); -console.log(sumWithInitial); -// Expected output: 10 +console.log(sumWithInitial); // Expected output: 10 ``` @@ -457,35 +374,23 @@ You can learn more about `array.reduce` from
-> Expected CLI logs +> -``` -[CLI] ✔ Compiled job from getPosts.js -GET request succeeded with 200 ✓ -[R/T] ✔ Operation 1 complete in 825ms -[R/T] ✔ Operation 2 complete in 0ms -[JOB] ℹ Post with userId 1 [ - //All of posts for userId 1 -] -[R/T] ✔ Operation 3 complete in 12ms -[CLI] ✔ Writing output to tmp/output.json -[CLI] ✔ Done in 1.239s! ✨ -``` +
+ Expand to see expected CLI logs -#### 🏆 Challenge: extract names & emails +``` -Using -[https://jsonplaceholder.typicode.com/posts/1/comments](https://jsonplaceholder.typicode.com/posts/1/comments) -API fetch comments for post with id 1 and extract name and email from each -comment in that post +[CLI] ✔ Compiled job from getPosts.js GET request succeeded with 200 ✓ [R/T] ✔ +Operation 1 complete in 825ms [R/T] ✔ Operation 2 complete in 0ms [JOB] ℹ Post +with userId 1 [ //All of posts for userId 1 ] [R/T] ✔ Operation 3 complete in +12ms [CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 1.239s! ✨ -1. Get post all comments for post id 1 -2. Extract name and email from comments -3. Log the extracted data from comments +``` -Discuss the results with your administrator. +
-### 6. Debugging errors +### 5. Debugging errors When debugging, it’s interesting to use log to have a visual representation of the content of the manipulated objects (such as state). @@ -506,6 +411,10 @@ fn(state => { ##### Create **debug.js** and paste the code below +
+ + Expand to see debug.js + ```jsx title="debug.js" // Get all posts get('posts'); @@ -519,22 +428,25 @@ fn(state => { }); ``` -##### Run **openfn debug.js -a http** +
-> Expected CLI logs +##### Run **openfn debug.js -a http** -```bash -[CLI] ✘ TypeError: path.match is not a function - at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26) - at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22) - at getPostbyIndex (vm:module(0):5:37) - at vm:module(0):18:36 - at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12 - at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26 - at process.processTicksAndRejections (node:internal/process/task_queues:95:5) - at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18) - at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20) -``` +
+ Expected CLI logs + ```bash + [CLI] ✘ TypeError: path.match is not a function + at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26) + at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22) + at getPostbyIndex (vm:module(0):5:37) + at vm:module(0):18:36 + at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12 + at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26 + at process.processTicksAndRejections (node:internal/process/task_queues:95:5) + at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18) + at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20) + ``` +
As you can see from our logs that helper function `dataValue` has a TypeError, To troubleshoot this you can go to the documentation for **dataValue -> @@ -545,36 +457,25 @@ According to the docs, dataValue take path which is a string type. But in our operation we were passing an integer, that’s why we have a _TypeError_. You can fix the error by passing a string in dataValue i.e `console.log(dataValue(“1”))` -> Expected CLI logs - -```bash -[CLI] ✔ Compiled job from debug.js -GET request succeeded with 200 ✓ -[R/T] ✔ Operation 1 complete in 722ms -[JOB] ℹ [Function (anonymous)] -[R/T] ✔ Operation 2 complete in 1ms -[CLI] ✔ Writing output to tmp/output.json -[CLI] ✔ Done in 1.102s ✨ -``` +
+ Expected CLI logs + ```bash + [CLI] ✔ Compiled job from debug.js + GET request succeeded with 200 ✓ + [R/T] ✔ Operation 1 complete in 722ms + [JOB] ℹ [Function (anonymous)] + [R/T] ✔ Operation 2 complete in 1ms + [CLI] ✔ Writing output to tmp/output.json + [CLI] ✔ Done in 1.102s ✨ + ``` +
If you need more information for debugging you can pass -l debug which will give all information about the run i.e `openfn debug.js -a http -l debug` -#### 🏆 Challenge: control error messages - -Debug what is causing an error on the following line of code and display the -error message - -```jsx -// Get post where id is 180 -get('posts/180'); -``` - -Discuss the results with your administrator. - -### 7. Each and array iteration +### 6. Each and array iteration We often have to perform the same operation multiple times for items in an array. Most of the helper functions for data manipulation are inherited from @@ -582,6 +483,9 @@ array. Most of the helper functions for data manipulation are inherited from ##### Modify getPosts.js to group posts by user-ID +
+Expand to see getPosts.js + ```js title="getPosts.js" // Get all posts get('posts'); @@ -615,6 +519,8 @@ each('posts[*]', state => { }); ``` +
+ Notice how this code uses the `each` function, a helper function defined in [language-common](/adaptors/packages/common-docs/#eachdatasource-operation--operation) but accessed from this job that is using language-http. Most adaptors import and @@ -622,35 +528,24 @@ export many functions from `language-common`. ##### Run **openfn getPosts.js -a http -o tmp/output.json** -> Expected CLI logs - -```bash -[CLI] ✔ Compiled job from getPosts.js -GET request succeeded with 200 ✓ -[R/T] ✔ Operation 1 complete in 730ms -[R/T] ✔ Operation 2 complete in 0ms -[R/T] ✔ Operation 3 complete in 0ms -[JOB] ℹ Posts [ -// Posts -] -[R/T] ✔ Operation 4 complete in 10ms -[CLI] ✔ Writing output to tmp/output.json -[CLI] ✔ Done in 1.091s! ✨ -``` - -#### 🏆 Challenge: Reduce, filter, and map - -Using Javascript globals i.e `Array.reduce`, `Array.filter` or `Array.map`, -build function that will get posts by user id. - -1. Create a file called job1.js -2. Add the 1st operation which is get all posts -3. Add 2nd operation which has a function that filter posts by id -4. Use the function from 2nd operation to get all post for user id 1 - -Discuss the results with your administrator. +
+ Expand to see expected CLI logs + ```bash + [CLI] ✔ Compiled job from getPosts.js + GET request succeeded with 200 ✓ + [R/T] ✔ Operation 1 complete in 730ms + [R/T] ✔ Operation 2 complete in 0ms + [R/T] ✔ Operation 3 complete in 0ms + [JOB] ℹ Posts [ + // Posts + ] + [R/T] ✔ Operation 4 complete in 10ms + [CLI] ✔ Writing output to tmp/output.json + [CLI] ✔ Done in 1.091s! ✨ + ``` +
-### 8. Running Workflows +### 7. Running Workflows As of `v0.0.35` the `@openfn/cli` supports running not only jobs, but also _workflows_. Running a workflow allows you to define a list of jobs and rules From 38a4bf430de4505f5ee7ddb3b5575c0af2d0392b Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 18 Dec 2023 17:33:44 +0300 Subject: [PATCH 10/14] update sidebar menu --- sidebars-main.js | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sidebars-main.js b/sidebars-main.js index c31cdabb374..61e7b0559d8 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -40,7 +40,8 @@ module.exports = { items: [ 'build-for-developers/cli-intro', 'build-for-developers/cli-usage', - 'build-for-developers/cli-tutorial', + 'build-for-developers/cli-walkthrough', + 'build-for-developers/cli-challenges', 'build-for-developers/build-with-api', 'build-for-developers/security-for-devs', //'build-for-developers/for-devs' From 87c6981675a9b2393604bbb7a2167c85d6661652 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Mon, 18 Dec 2023 17:44:08 +0300 Subject: [PATCH 11/14] fix broken link --- docs/build-for-developers/cli-usage.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/build-for-developers/cli-usage.md b/docs/build-for-developers/cli-usage.md index 7e88d41bc0b..3ddaa576e23 100644 --- a/docs/build-for-developers/cli-usage.md +++ b/docs/build-for-developers/cli-usage.md @@ -146,5 +146,5 @@ openfn repo clean openfn path/to/workflow.json -i ``` -Check out this detailed [tutorial](cli-tutorial#8-running-workflows) on running -workflows via the CLI. +Check out this detailed [tutorial](cli-walkthrough#7-running-workflows) on +running workflows via the CLI. From e2caafe8f26492322e48145963544760b2c6fca4 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 19 Dec 2023 12:03:19 +0300 Subject: [PATCH 12/14] improve challenges --- docs/build-for-developers/cli-challenges.md | 300 +++++++++++++------ docs/build-for-developers/cli-walkthrough.md | 11 + 2 files changed, 214 insertions(+), 97 deletions(-) diff --git a/docs/build-for-developers/cli-challenges.md b/docs/build-for-developers/cli-challenges.md index 377eea728b5..47460fd9e8b 100644 --- a/docs/build-for-developers/cli-challenges.md +++ b/docs/build-for-developers/cli-challenges.md @@ -4,130 +4,202 @@ sidebar_label: CLI Challenges slug: /cli-challenges --- -#### 🏆 Challenge: Write a job that prints your name - -1. Modify `hello.js` to print your name. -2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`. -3. Validate that you receive the logs below: - -```bash -[CLI] ✔ Compiled job from hello.js -[JOB] ℹ My name is { YourName } -[R/T] ✔ Operation 1 complete in 0ms -[CLI] ✔ Writing output to tmp/output.json -[CLI] ✔ Done in 366ms! ✨ -``` - ---- +#### Solve real-world problems and showcase your command-line skills by participating in our CLI challenges. -#### 🏆 Challenge: Get and inspect data via HTTP +:::tip Important Notes -Using the -[https://jsonplaceholder.typicode.com/users](https://jsonplaceholder.typicode.com/users) -API, get a list of users and print the first user object. +- A developer with a bit of Javascript experience should be able to write, run, + and debug complex, multi-step jobs with OpenFn, using nothing but a text + editor and their terminal. +- If you are stuck and need help, please post in + [community.openfn.org](community.openfn.org). +
+ Expand to see bug report template -1. Create file called `getUsers.js` and write your operation to fetch the user. -2. Run the job using the OpenFn/cli - `openfn getUsers.js -a http -o tmp/output.json`. -3. Validate that you receive this expected CLI logs: + ``` -```bash -openfn getUsers.js -a http -o tmp/output.json -``` + Subject: Bug Report - [Brief Description] -
- Expand to see expected CLI logs + **Description:** + [Concise description of the bug.] -``` -[CLI] ✔ Compiled job from hello.js GET request succeeded with 200 ✓ -[R/T] ✔ Operation 1 complete in 581ms -[JOB] ℹ { - id: 1, - name: 'Leanne Graham', - username: 'Bret', - email: 'Sincere@april.biz', - address: { - street: 'Kulas Light', - suite: 'Apt. 556', - city: 'Gwenborough', - zipcode: '92998-3874', - geo: { lat: '-37.3159', lng: '81.1496' } - }, - phone: '1-770-736-8031 x56442', - website: 'hildegard.org', - company: { - name: 'Romaguera-Crona', - catchPhrase: 'Multi-layered client-server neural-net', - bs: 'harness real-time e-markets' - } -} -[R/T] ✔ Operation 2 complete in 2ms -[CLI] ✔ Writing output to tmp/output.json [CLI] ✔ Done in 950ms! ✨ -``` + **Steps to Reproduce:** + 1. + 2. + 3. -
+ **Environment:** + - OS: [e.g., Windows 10] + - CLI: [e.g., v0.4.11] + - Node: [e.g., v 18.17.1] + - NPM: [e.g., 8.19.2] -:::warning Placeholder Data + **Attachments:** + [Screenshots, error messages, or relevant files.] -The data displayed in this CLI logs is generated from a -[JSONPlaceholder](https://jsonplaceholder.typicode.com/) API and does not -represent real-world information. It is intended for testing and development -purposes only. + ``` -For accurate testing, consider using real data from your API or service. +
::: +### 🏆 Create personalized greeting + +**Overview:** + +Create a new `hello.js` job to display a personalized greeting with your name. + +**Objective:** + +Compose a OpenFn job using [common adaptor](/adaptors/packages/common-docs) that +outputs a greeting message containing your name. + +**Requirements:** + +1. Install the latest version of common adaptor. + + ``` + openfn repo install @openfn/language-common + ``` + +**Tasks:** + +1. Create a new file named `hello.js`. +2. Write a JavaScript script in `hello.js` to generate a greeting with your + name. +3. Run the job using the command `openfn hello.js -a common -o tmp/output.json`. +4. Confirm the successful execution. + +**Review Checklist:** + +- [ ] Successfully created a new file `hello.js`. +- [ ] Wrote a JavaScript script in `hello.js` for a personalized greeting. +- [ ] Executed the job using the provided command. +- [ ] Verified the correct logs in the CLI output. + --- -#### 🏆 Challenge: Fetch Covid-19 metadata +### 🏆 Fetch and inspect data via HTTP -1. Using the [disease.sh API](https://disease.sh/), write an operation that - returns all covid-19 metadata. +**Overview:** -:::tip +Write a job to fetch user data from the +[JSONPlaceholder API](https://jsonplaceholder.typicode.com/users) using OpenFn +[http adaptor](/adaptors/packages/http-docs). -`https://disease.sh/v3/covid-19/` as your **baseUrl** in `state.configuration` +**Objective:** -::: +Fetch and print the details of the first user from the JSONPlaceholder API. + +**Requirements:** + +1. Utilize the + [JSONPlaceholder API](https://jsonplaceholder.typicode.com/users). +2. Create a file named `getUsers.js` to contain the script. + +**Tasks:** + +1. Create a file (`getUsers.js`) to house the script. +2. Fetch a list of users from the JSONPlaceholder API. +3. Print the details of the first user. +4. Run the job using OpenFn/cli: + `openfn getUsers.js -a http -o tmp/output.json`. +5. Validate the expected CLI logs. + +**Review Checklist:** -2. Validate your output: there are a lot of ways you might choose to format or - display this data. Share your results with your administrator for feedback. +- [ ] Successful retrieval of user data. +- [ ] Correct printing of the first user's details. +- [ ] Proper use of the OpenFn http adaptor functions. +- [ ] Verified the correct logs in the CLI output. --- -#### 🏆 Challenge: Practice CLI arguments and commands +### 🏆 Retrieve Covid-19 metadata + +**Overview:** + +Fetch and present COVID-19 metadata using the +[disease.sh API](https://disease.sh/). + +**Objective:** + +Write a job that retrieves comprehensive COVID-19 data from the API and group it +by region. + +**Requirements:** + +1. Install the latest version of http adaptor. + +``` +openfn repo install @openfn/language-http +``` -Perform these tasks and submit answers to the discussion questions to your -administrator for feedback. +**Tasks:** -1. Compile a openfn job (**hello.js**). +1. Write an OpenFn operation to pull COVID-19 metadata from the + [disease.sh API](https://disease.sh/). + - Utilize `https://disease.sh/v3/covid-19/` as your **baseUrl** in + `state.configuration`. +2. Run the job using the OpenFn CLI with the command + `openfn your_operation_file.js -a http -o tmp/output.json`. +3. Evaluate the output and explore different ways to format or present the + COVID-19 data by region. - > What's the difference between the job you wrote and the compiled job? +**Review Checklist:** -2. Run a job with the log level set to `none`, and then run it again with the - log level set to `debug`. +- [ ] Successfully created an OpenFn operation file. +- [ ] Implemented code to retrieve COVID-19 metadata from the provided API. +- [ ] Executed the job using the provided CLI command. +- [ ] Explored various formatting or display options for the obtained data. - > When is it appropriate to use these different log levels? +> Feel free to experiment with the data presentation to enhance your +> understanding. Good luck! 🌐🦠 --- -#### 🏆 Challenge: extract names & emails +### 🏆 Extract names & emails + +**Overview:** + +In this challenge, you will use the JSONPlaceholder API to fetch comments for a +specific post (post ID 1). Your task is to extract the "name" and "email" fields +from each comment and log the extracted data. + +**Objective:** + +Write a job that retrieves comments for post ID 1, extracts the "name" and +"email" fields from each comment, and logs the extracted data. + +**Requirements:** + +- Basic knowledge of JavaScript. +- OpenFn CLI installed on your machine. -Using -[https://jsonplaceholder.typicode.com/posts/1/comments](https://jsonplaceholder.typicode.com/posts/1/comments) -API fetch comments for post with id 1 and extract name and email from each -comment in that post +**Tasks:** -1. Get post all comments for post id 1 -2. Extract name and email from comments -3. Log the extracted data from comments +1. **Get Post Comments:** -Discuss the results with your administrator. + - Add an operation to fetch all comments for the post with ID 1 from the + [JSONPlaceholder API](https://jsonplaceholder.typicode.com/posts/1/comments). + +2. **Extract Name and Email:** + + - Write a function to extract the "name" and "email" fields from each + comment. + +3. **Log Extracted Data:** + - Log the extracted data (name and email) from each comment to the console. + +**Review Checklist:** + +- [ ] Successfully fetched comments for post ID 1. +- [ ] Implemented a function to extract "name" and "email" from comments. +- [ ] Logged the extracted data to the console. --- -#### 🏆 Challenge: control error messages +### 🏆 Control error messages Debug what is causing an error on the following line of code and display the error message @@ -137,18 +209,52 @@ error message get('posts/180'); ``` -Discuss the results with your administrator. - --- -#### 🏆 Challenge: Reduce, filter, and map +### 🏆 Data transformation and cleaning + +**Overview:** + +In this challenge, you will use JavaScript global array methods, specifically +`Array.reduce`, `Array.filter`, or `Array.map`, to create a series of operations +that fetch and filter posts by user ID. + +**Objective:** + +Write a job that retrieves posts by a specified user ID `1` + +**Requirements:** + +1. Utilize JSONPlaceholder API `https://jsonplaceholder.typicode.com`. +2. Install the latest version of http adaptor. + +``` +openfn repo install @openfn/language-http +``` + +**Tasks:** + +1. **Create File:** + + - Create a file named `getPosts.js` for your job. + +2. **Get All Posts:** + + - Add the first operation to fetch all posts. Use the provided API or any + other source of your choice that provides a list of posts. + +3. **Filter Posts by ID:** + + - Add a second operation with a function that filters posts by user ID. You + can use `Array.filter` or any other suitable method for this task. + +4. **Fetch Posts for User ID 1:** -Using Javascript globals i.e `Array.reduce`, `Array.filter` or `Array.map`, -build function that will get posts by user id. + - Use the function from the second operation to filter posts for user ID 1. -1. Create a file called job1.js -2. Add the 1st operation which is get all posts -3. Add 2nd operation which has a function that filter posts by id -4. Use the function from 2nd operation to get all post for user id 1 +**Review Checklist:** -Discuss the results with your administrator. +- [ ] Created `getPosts.js` file. +- [ ] Successfully fetched all posts. +- [ ] Implemented a function to filter posts by user ID. +- [ ] Retrieved posts for user ID 1. diff --git a/docs/build-for-developers/cli-walkthrough.md b/docs/build-for-developers/cli-walkthrough.md index 23bf275c0b8..679f0c539cb 100644 --- a/docs/build-for-developers/cli-walkthrough.md +++ b/docs/build-for-developers/cli-walkthrough.md @@ -149,6 +149,17 @@ adaptor using `-i` argument
+:::warning Placeholder Data + +The data displayed in this CLI logs is generated from a +[JSONPlaceholder](https://jsonplaceholder.typicode.com/) API and does not +represent real-world information. It is intended for testing and development +purposes only. + +For accurate testing, consider using real data from your API or service. + +::: + ### 3. Understanding `state` If a job expression is a set of instructions for a chef (a recipe?) then the From 313961bb5956da9281932185b3c900902e4ede0d Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 19 Dec 2023 12:08:06 +0300 Subject: [PATCH 13/14] update community link --- docs/build-for-developers/cli-challenges.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/build-for-developers/cli-challenges.md b/docs/build-for-developers/cli-challenges.md index 47460fd9e8b..b9d185dd456 100644 --- a/docs/build-for-developers/cli-challenges.md +++ b/docs/build-for-developers/cli-challenges.md @@ -12,7 +12,7 @@ slug: /cli-challenges and debug complex, multi-step jobs with OpenFn, using nothing but a text editor and their terminal. - If you are stuck and need help, please post in - [community.openfn.org](community.openfn.org). + [community.openfn.org](https://community.openfn.org).
Expand to see bug report template From df4f9f7d09db239756f76d759b71abe6d6619b37 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 19 Dec 2023 15:10:29 +0300 Subject: [PATCH 14/14] improve running workflows section --- docs/build-for-developers/cli-walkthrough.md | 44 +++++++++++--------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/build-for-developers/cli-walkthrough.md b/docs/build-for-developers/cli-walkthrough.md index 679f0c539cb..d3e31befa66 100644 --- a/docs/build-for-developers/cli-walkthrough.md +++ b/docs/build-for-developers/cli-walkthrough.md @@ -324,7 +324,7 @@ Since we have update our configuration in our `state.json` we can now use
-### 4. Transform data in a sequence of operations +### 4. Clean & Transform Data In most cases you need to manipulate, clean, or transform data at some step in your workflow. For example after we get data from the @@ -583,25 +583,29 @@ initial state. A workflow is the execution plan for running several jobs in a sequence. It is defined as a JSON object that consists of the following properties: -- `start` (optional): The ID of the job that should be executed first (defaults - to jobs[0]). -- `jobs` (required): An array of job objects, each of which represents a - specific task to be executed. - - `id` (required): A job name that is unique to the workflow and helps you ID - your job. - - `configuration`: (optional) Specifies the configuration file associated with - the job. - - `data` (optional): A JSON object that contains the pre-populated data. - - `adaptor` (required): Specifies the adaptor used for the job (version - optional). - - `expression` (required): Specifies the JavaScript file associated with the - job. It can also be a string that contains a JavaScript function to be - executed as the job. - - `next` (optional): An object that specifies which jobs to call next. All - edges returning true will run. The object should have one or more key-value - pairs, where the key is the ID of the next job, and the value is a boolean - expression that determines whether the next job should be executed.If there - are no next edges, the workflow will end. +```json +{ + "start": "a", // optionally specify the start node (defaults to jobs[0]) + "jobs": [ + { + "id": "a", + "expression": "fn((state) => state)", // code or a path + "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional) + "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state) + "configuration": {}, // Use this to pass credentials + "next": { + // This object defines which jobs to call next + // All edges returning true will run + // If there are no next edges, the workflow will end + "b": true, + "c": { + "condition": "!state.error" // Note that this is an expression, not a function + } + } + } + ] +} +``` ###### Example of a workflow