diff --git a/docs/build-for-developers/cli-challenges.md b/docs/build-for-developers/cli-challenges.md
new file mode 100644
index 00000000000..b9d185dd456
--- /dev/null
+++ b/docs/build-for-developers/cli-challenges.md
@@ -0,0 +1,260 @@
+---
+title: CLI Challenges
+sidebar_label: CLI Challenges
+slug: /cli-challenges
+---
+
+#### Solve real-world problems and showcase your command-line skills by participating in our CLI challenges.
+
+:::tip Important Notes
+
+- A developer with a bit of Javascript experience should be able to write, run,
+ and debug complex, multi-step jobs with OpenFn, using nothing but a text
+ editor and their terminal.
+- If you are stuck and need help, please post in
+ [community.openfn.org](https://community.openfn.org).
+
+ Expand to see bug report template
+
+ ```
+
+ Subject: Bug Report - [Brief Description]
+
+ **Description:**
+ [Concise description of the bug.]
+
+ **Steps to Reproduce:**
+ 1.
+ 2.
+ 3.
+
+ **Environment:**
+ - OS: [e.g., Windows 10]
+ - CLI: [e.g., v0.4.11]
+ - Node: [e.g., v 18.17.1]
+ - NPM: [e.g., 8.19.2]
+
+ **Attachments:**
+ [Screenshots, error messages, or relevant files.]
+
+ ```
+
+
+
+:::
+
+### π Create personalized greeting
+
+**Overview:**
+
+Create a new `hello.js` job to display a personalized greeting with your name.
+
+**Objective:**
+
+Compose a OpenFn job using [common adaptor](/adaptors/packages/common-docs) that
+outputs a greeting message containing your name.
+
+**Requirements:**
+
+1. Install the latest version of common adaptor.
+
+ ```
+ openfn repo install @openfn/language-common
+ ```
+
+**Tasks:**
+
+1. Create a new file named `hello.js`.
+2. Write a JavaScript script in `hello.js` to generate a greeting with your
+ name.
+3. Run the job using the command `openfn hello.js -a common -o tmp/output.json`.
+4. Confirm the successful execution.
+
+**Review Checklist:**
+
+- [ ] Successfully created a new file `hello.js`.
+- [ ] Wrote a JavaScript script in `hello.js` for a personalized greeting.
+- [ ] Executed the job using the provided command.
+- [ ] Verified the correct logs in the CLI output.
+
+---
+
+### π Fetch and inspect data via HTTP
+
+**Overview:**
+
+Write a job to fetch user data from the
+[JSONPlaceholder API](https://jsonplaceholder.typicode.com/users) using OpenFn
+[http adaptor](/adaptors/packages/http-docs).
+
+**Objective:**
+
+Fetch and print the details of the first user from the JSONPlaceholder API.
+
+**Requirements:**
+
+1. Utilize the
+ [JSONPlaceholder API](https://jsonplaceholder.typicode.com/users).
+2. Create a file named `getUsers.js` to contain the script.
+
+**Tasks:**
+
+1. Create a file (`getUsers.js`) to house the script.
+2. Fetch a list of users from the JSONPlaceholder API.
+3. Print the details of the first user.
+4. Run the job using OpenFn/cli:
+ `openfn getUsers.js -a http -o tmp/output.json`.
+5. Validate the expected CLI logs.
+
+**Review Checklist:**
+
+- [ ] Successful retrieval of user data.
+- [ ] Correct printing of the first user's details.
+- [ ] Proper use of the OpenFn http adaptor functions.
+- [ ] Verified the correct logs in the CLI output.
+
+---
+
+### π Retrieve Covid-19 metadata
+
+**Overview:**
+
+Fetch and present COVID-19 metadata using the
+[disease.sh API](https://disease.sh/).
+
+**Objective:**
+
+Write a job that retrieves comprehensive COVID-19 data from the API and group it
+by region.
+
+**Requirements:**
+
+1. Install the latest version of http adaptor.
+
+```
+openfn repo install @openfn/language-http
+```
+
+**Tasks:**
+
+1. Write an OpenFn operation to pull COVID-19 metadata from the
+ [disease.sh API](https://disease.sh/).
+ - Utilize `https://disease.sh/v3/covid-19/` as your **baseUrl** in
+ `state.configuration`.
+2. Run the job using the OpenFn CLI with the command
+ `openfn your_operation_file.js -a http -o tmp/output.json`.
+3. Evaluate the output and explore different ways to format or present the
+ COVID-19 data by region.
+
+**Review Checklist:**
+
+- [ ] Successfully created an OpenFn operation file.
+- [ ] Implemented code to retrieve COVID-19 metadata from the provided API.
+- [ ] Executed the job using the provided CLI command.
+- [ ] Explored various formatting or display options for the obtained data.
+
+> Feel free to experiment with the data presentation to enhance your
+> understanding. Good luck! ππ¦
+
+---
+
+### π Extract names & emails
+
+**Overview:**
+
+In this challenge, you will use the JSONPlaceholder API to fetch comments for a
+specific post (post ID 1). Your task is to extract the "name" and "email" fields
+from each comment and log the extracted data.
+
+**Objective:**
+
+Write a job that retrieves comments for post ID 1, extracts the "name" and
+"email" fields from each comment, and logs the extracted data.
+
+**Requirements:**
+
+- Basic knowledge of JavaScript.
+- OpenFn CLI installed on your machine.
+
+**Tasks:**
+
+1. **Get Post Comments:**
+
+ - Add an operation to fetch all comments for the post with ID 1 from the
+ [JSONPlaceholder API](https://jsonplaceholder.typicode.com/posts/1/comments).
+
+2. **Extract Name and Email:**
+
+ - Write a function to extract the "name" and "email" fields from each
+ comment.
+
+3. **Log Extracted Data:**
+ - Log the extracted data (name and email) from each comment to the console.
+
+**Review Checklist:**
+
+- [ ] Successfully fetched comments for post ID 1.
+- [ ] Implemented a function to extract "name" and "email" from comments.
+- [ ] Logged the extracted data to the console.
+
+---
+
+### π Control error messages
+
+Debug what is causing an error on the following line of code and display the
+error message
+
+```jsx
+// Get post where id is 180
+get('posts/180');
+```
+
+---
+
+### π Data transformation and cleaning
+
+**Overview:**
+
+In this challenge, you will use JavaScript global array methods, specifically
+`Array.reduce`, `Array.filter`, or `Array.map`, to create a series of operations
+that fetch and filter posts by user ID.
+
+**Objective:**
+
+Write a job that retrieves posts by a specified user ID `1`
+
+**Requirements:**
+
+1. Utilize JSONPlaceholder API `https://jsonplaceholder.typicode.com`.
+2. Install the latest version of http adaptor.
+
+```
+openfn repo install @openfn/language-http
+```
+
+**Tasks:**
+
+1. **Create File:**
+
+ - Create a file named `getPosts.js` for your job.
+
+2. **Get All Posts:**
+
+ - Add the first operation to fetch all posts. Use the provided API or any
+ other source of your choice that provides a list of posts.
+
+3. **Filter Posts by ID:**
+
+ - Add a second operation with a function that filters posts by user ID. You
+ can use `Array.filter` or any other suitable method for this task.
+
+4. **Fetch Posts for User ID 1:**
+
+ - Use the function from the second operation to filter posts for user ID 1.
+
+**Review Checklist:**
+
+- [ ] Created `getPosts.js` file.
+- [ ] Successfully fetched all posts.
+- [ ] Implemented a function to filter posts by user ID.
+- [ ] Retrieved posts for user ID 1.
diff --git a/docs/build-for-developers/cli-intro.md b/docs/build-for-developers/cli-intro.md
index c4ff1120aaa..1f1583439d3 100644
--- a/docs/build-for-developers/cli-intro.md
+++ b/docs/build-for-developers/cli-intro.md
@@ -1,49 +1,146 @@
---
-title: Intro to the OpenFn CLI
-sidebar_label: Intro to CLI
+title: Get started with the OpenFn CLI
+sidebar_label: Get started
slug: /cli
---
-:::info What is this tutorial?
+#### Build and test your automated workflows and integrations via the command line.
-- It's a _hands-on_ way to learn about the new OpenFn CLI. By following the
- prompts and "challenges", a developer with a bit of Javascript experience
- should be able to write, run, and debug complex, multi-step jobs with OpenFn,
- using nothing but a text editor and their terminal.
-- The estimated time to finish this developer challenge is 1 to 2 hours
- (depending on your familiarity with the underlying concepts and tooling)
-- If you are stuck and need help, please post in
- [community.openfn.org](https://community.openfn.org/t/about-the-job-writing-category/11/1)
+The OpenFn CLI is a developer tool to help you build, test, and manage your
+workflows and integration with OpenFn directly from the command line. Itβs
+simple to install, works on macOS, Windows, and Linux, and offers a range of
+functionality to enhance your developer experience with OpenFn. You can use the
+OpenFn CLI to:
-:::
+- Securely run OpenFn jobs and workflows
+- Troubleshoot and debug OpenFn jobs
+- Deployment of workflows to OpenFn
-## Intro to the OpenFn CLI
+---
+
+### Before you start
+
+Before you begin with the @openfn/cli, make sure to follow these simple steps:
+
+1. **Code Editor:** Ensure you have a code editor installed on your machine. You
+ can use popular editors like [VS Code](https://code.visualstudio.com/) or
+ [Sublime](https://www.sublimetext.com/).
+2. **Node.js Installation:** Install Node.js (version 18 or later): - For Linux,
+ Windows, or macOS, use a version manager like
+ [nvm](https://github.com/nvm-sh/nvm) or
+ [asdf](https://asdf-vm.com/guide/getting-started.html). -
+ [Install Node.js](https://kinsta.com/blog/how-to-install-node-js/) by
+ following this guide.
+3. **Understand OpenFn Basics:** Have a basic understanding of OpenFn,
+ particularly jobs and adaptors. Check out the
+ [Intro section](/documentation/next) on this site.
+
+---
+
+### Install the OpenFn CLI
+
+To download the latest version of
+[@openfn/cli](https://www.npmjs.com/package/@openfn/cli), on the command line,
+run the following command.
+
+```bash
+npm install -g @openfn/cli
+```
+
+**Make sure everything works by running the built-in test job**
+
+```bash
+openfn test
+```
-The [@openfn/cli](https://github.com/OpenFn/kit/tree/main/packages/cli) is a
-command line interface for running OpenFn workflows locally. It enables
-developers to run, build, and test steps in an OpenFn workflow.
+The word `openfn` will invoke the CLI. The word `test` will invoke the test
+command.
-This CLI replaces [@openfn/devtools](https://github.com/OpenFn/devtools) and
-provides a new suite of features and improvements, including:
+Expand to see the expected output.
-- a new runtime and compiler for executing and creating runnable OpenFn jobs,
-- customizable logging output,
-- automatic installation of language adaptors,
-- and support for the adaptors monorepo
- ([@openfn/adaptors](https://github.com/OpenFn/adaptors)) where all OpenFn
- adaptor source code and documentation lives.
+ [CLI] βΉ Versions:
+ βΈ node.js 18.12.1
+ βΈ cli 0.4.11
+ βΈ runtime 0.2.2
+ βΈ compiler 0.0.38
+ [CLI] βΉ Running test job...
+ [CLI] βΉ Workflow object:
+ [CLI] βΉ {
+ "start": "start",
+ "jobs": [
+ {
+ "id": "start",
+ "data": {
+ "defaultAnswer": 42
+ },
+ "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()",
+ "next": {
+ "calculate": "!state.error"
+ }
+ },
+ {
+ "id": "calculate",
+ "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()",
+ "next": {
+ "result": true
+ }
+ },
+ {
+ "id": "result",
+ "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()"
+ }
+ ]
+ }
-These features are designed to make it easier and more convenient for developers
-to use and understand OpenFn.
+ [CLI] β Compilation complete
+ [R/T] β¦ Starting job start
+ [JOB] βΉ Starting computer...
+ [R/T] βΉ Operation 1 complete in 0ms
+ [R/T] β Completed job start in 1ms
+ [R/T] β¦ Starting job calculate
+ [JOB] βΉ Calculating to life, the universe, and everything..
+ [R/T] βΉ Operation 1 complete in 0ms
+ [R/T] β Completed job calculate in 1ms
+ [R/T] β¦ Starting job result
+ [R/T] βΉ Operation 1 complete in 0ms
+ [R/T] β Completed job result in 0ms
+ [CLI] β Result: 42
-:::caution Looking for a way to execute jobs from OpenFn v1 locally? Use Core!
+
+ A workflow has a structure like this
+
+```json
+{
+ "start": "a", // optionally specify the start node (defaults to jobs[0])
+ "jobs": [
+ {
+ "id": "a",
+ "expression": "fn((state) => state)", // code or a path
+ "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional)
+ "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state)
+ "configuration": {}, // Use this to pass credentials
+ "next": {
+ // This object defines which jobs to call next
+ // All edges returning true will run
+ // If there are no next edges, the workflow will end
+ "b": true,
+ "c": {
+ "condition": "!state.error" // Note that this is an expression, not a function
+ }
+ }
+ }
+ ]
+}
+```
+
+
+
+**To run a workflow:**
+
+```bash
+openfn path/to/workflow.json -i
+```
-Learn more about CLI
-[github.com/OpenFn/kit/](https://github.com/OpenFn/kit/tree/main/packages/cli)
+Check out this detailed [tutorial](cli-walkthrough#7-running-workflows) on
+running workflows via the CLI.
diff --git a/docs/build-for-developers/cli-tutorial.md b/docs/build-for-developers/cli-walkthrough.md
similarity index 59%
rename from docs/build-for-developers/cli-tutorial.md
rename to docs/build-for-developers/cli-walkthrough.md
index 58e925fe8e6..d3e31befa66 100644
--- a/docs/build-for-developers/cli-tutorial.md
+++ b/docs/build-for-developers/cli-walkthrough.md
@@ -1,114 +1,11 @@
---
-title: CLI Walkthrough & Challenges
-sidebar_label: CLI Tutorial
-slug: /cli-tutorial
+title: CLI Walkthrough
+sidebar_label: CLI Walkthrough
+slug: /cli-walkthrough
---
-
-## Walkthrough & Challenges
-
### 1. Getting started with the CLI
-Let's start by running a simple command with the CLI. Type the following into
-your terminal:
-
-```bash
-openfn test
-```
-
-The word `openfn` will invoke the CLI. The word `test` will invoke the test
-command.
-
-
- You should see some output like this:
-
-```bash
-[CLI] βΉ Versions:
- βΈ node.js 18.12.1
- βΈ cli 0.0.39
- βΈ runtime 0.0.24
- βΈ compiler 0.0.32
-[CLI] βΉ Running test job...
-[CLI] βΉ Workflow object:
-[CLI] βΉ {
- "start": "start",
- "jobs": [
- {
- "id": "start",
- "data": {
- "defaultAnswer": 42
- },
- "expression": "const fn = () => (state) => { console.log('Starting computer...'); return state; }; fn()",
- "next": {
- "calculate": "!state.error"
- }
- },
- {
- "id": "calculate",
- "expression": "const fn = () => (state) => { console.log('Calculating to life, the universe, and everything..'); return state }; fn()",
- "next": {
- "result": true
- }
- },
- {
- "id": "result",
- "expression": "const fn = () => (state) => ({ data: { answer: state.data.answer || state.data.defaultAnswer } }); fn()"
- }
- ]
-}
-
-[CLI] β Compilation complete
-[R/T] β¦ Starting job start
-[JOB] βΉ Starting computer...
-[R/T] βΉ Operation 1 complete in 0ms
-[R/T] β Completed job start in 1ms
-[R/T] β¦ Starting job calculate
-[JOB] βΉ Calculating to life, the universe, and everything..
-[R/T] βΉ Operation 1 complete in 0ms
-[R/T] β Completed job calculate in 1ms
-[R/T] β¦ Starting job result
-[R/T] βΉ Operation 1 complete in 0ms
-[R/T] β Completed job result in 0ms
-[CLI] β Result: 42
-
-```
-
-
-
-What we've just done is executed a JavaScript expression, which we call a _job_.
-The output prefixed with `[JOB]` comes directly from `console.log` statements in
-our job code. All other output is the CLI trying to tell us what it is doing.
-
-
-What is a job?
-A job is Javascript code which follows a particular set of conventions.
-Typically a job has one or more operations which perform a particular
-task (like pulling information from a database, creating a record, etc.) and
-return state for the next operation to use.
-
-The test job we just ran looks like this:
-
-```js
-const fn = () => state => {
- console.log(
- 'Calculating the answer to life, the universe, and everything...'
- );
- return state * 2;
-};
-export default [fn()];
-```
-
-You can see this (and a lot more detail) by running the test command with
-debug-level logging:
-
-```bash
-openfn test --log debug
-```
-
-
-
-#### Tasks:
-
:::info To get started with @openfn/cli
1. Create a new folder for the repository you'll be working on by running the
@@ -135,61 +32,54 @@ openfn test --log debug
:::
-1. Create a file called `hello.js` and write the following code.
+1. Create a job file called `hello.js` and write the following code.
```js
console.log('Hello World!');
```
+
+ What is a job?
+ A job is Javascript code which follows a particular set of conventions.
+ Typically a job has one or more operations which perform a particular
+ task (like pulling information from a database, creating a record, etc.) and
+ return state for the next operation to use.
+
+
What is console.log?
console.log is a core JavaScript language function which lets
us send messages to the terminal window.
-1. Run the job using the CLI
-
- ```bash
- openfn hello.js -o tmp/output.json
- ```
+2. Run the job using the CLI
-
+ ```bash
+ openfn hello.js -o tmp/output.json
+ ```
+
View expected output
- ```bash
- [CLI] β WARNING: No adaptor provided!
- [CLI] β This job will probably fail. Pass an adaptor with the -a flag, eg:
- openfn job.js -a common
- [CLI] β Compiled from helo.js
- [R/T] β¦ Starting job job-1
- [JOB] βΉ Hello World!
- [R/T] β Completed job job-1 in 1ms
- [CLI] β State written to tmp/output.json
- [CLI] β Finished in 17ms β¨
+ ```bash
+ [CLI] β WARNING: No adaptor provided!
+ [CLI] β This job will probably fail. Pass an adaptor with the -a flag, eg:
+ openfn job.js -a common
+ [CLI] β Compiled from hello.js
+ [R/T] β¦ Starting job job-1
+ [JOB] βΉ Hello World!
+ [R/T] β Completed job job-1 in 1ms
+ [CLI] β State written to tmp/output.json
+ [CLI] β Finished in 17ms β¨
- ```
+ ```
-
+
Note that our `console.log` statement was printed as `[JOB] Hello world!`. Using
the console like this is helpful for debugging and/or understanding what's
happening inside our jobs.
-#### π Challenge: Write a job that prints your name
-
-1. Modify `hello.js` to print your name.
-2. Re-run the job by running `openfn hello.js -a common -o tmp/output.json`.
-3. Validate that you receive the logs below:
-
-```bash
-[CLI] β Compiled job from hello.js
-[JOB] βΉ My name is { YourName }
-[R/T] β Operation 1 complete in 0ms
-[CLI] β Writing output to tmp/output.json
-[CLI] β Done in 366ms! β¨
-```
-
### 2. Using adaptor helper functions
Adaptors are Javascript or Typescript modules that provide OpenFn users with a
@@ -233,7 +123,7 @@ Since it is our first time using the `http` adaptor, we are installing the
adaptor using `-i` argument
- 3. See expected CLI logs
+ 3. Expand to see expected CLI logs
```bash
[CLI] β Installing packages...
@@ -259,52 +149,16 @@ adaptor using `-i` argument
-#### π Challenge: Get and inspect data via HTTP
-
-Using the
-[https://jsonplaceholder.typicode.com/users](https://jsonplaceholder.typicode.com/users)
-API, get a list of users and print the first user object.
-
-1. Create file called `getUsers.js` and write your operation to fetch the user.
-2. Run the job using the OpenFn/cli
- `openfn getUsers.js -a http -o tmp/output.json`.
-3. Validate that you receive this expected CLI logs:
-
-```bash
-openfn getUsers.js -a http -o tmp/output.json
-```
+:::warning Placeholder Data
-
-See expected CLI logs:
+The data displayed in this CLI logs is generated from a
+[JSONPlaceholder](https://jsonplaceholder.typicode.com/) API and does not
+represent real-world information. It is intended for testing and development
+purposes only.
-```
-[CLI] β Compiled job from hello.js GET request succeeded with 200 β
-[R/T] β Operation 1 complete in 581ms
-[JOB] βΉ {
- id: 1,
- name: 'Leanne Graham',
- username: 'Bret',
- email: 'Sincere@april.biz',
- address: {
- street: 'Kulas Light',
- suite: 'Apt. 556',
- city: 'Gwenborough',
- zipcode: '92998-3874',
- geo: { lat: '-37.3159', lng: '81.1496' }
- },
- phone: '1-770-736-8031 x56442',
- website: 'hildegard.org',
- company: {
- name: 'Romaguera-Crona',
- catchPhrase: 'Multi-layered client-server neural-net',
- bs: 'harness real-time e-markets'
- }
-}
-[R/T] β Operation 2 complete in 2ms
-[CLI] β Writing output to tmp/output.json [CLI] β Done in 950ms! β¨
-```
+For accurate testing, consider using real data from your API or service.
-
+:::
### 3. Understanding `state`
@@ -314,7 +168,8 @@ bundle. See
["It all starts with stateβ"](/articles/2021/07/05/wrapping-my-head-around-jobs/#it-all-starts-with-state)
in the knowledge base for extra context.
-It usually looks something like this
+
+ It usually looks something like this
```json
{
@@ -334,6 +189,8 @@ It usually looks something like this
}
```
+
+
#### `state.configuration`
This key is where we put credentials which are used to authorize connections to
@@ -370,7 +227,8 @@ Specify a path to your `state.json` file with this command:
openfn hello.js -a http -s tmp/state.json -o tmp/output.json
```
-Expected CLI logs
+
+ Expand to see expected CLI logs
```
[CLI] β Compiled job from hello.js
@@ -381,6 +239,8 @@ GET request succeeded with 200 β
[CLI] β Done in 1.222s! β¨
```
+
+
#### How can we use state?
Each adaptor has a configuration schema that's recommended for use in your
@@ -399,20 +259,28 @@ of how to set up `state.configuration` for `language-http`.
1. Update your `state.json` to look like this:
- ```json title=state.json
- {
- "configuration": {
- "baseUrl": "https://jsonplaceholder.typicode.com"
- }
- }
- ```
+
+ Expand to see state.json
+
+ ```json title=state.json
+ {
+ "configuration": {
+ "baseUrl": "https://jsonplaceholder.typicode.com"
+ }
+ }
+ ```
- Since we have update our configuration in our `state.json` we can now use
- `get()` helper function without the need to specify the **baseUrl**βi.e
- `get('posts')`
+
+
+Since we have update our configuration in our `state.json` we can now use
+`get()` helper function without the need to specify the **baseUrl**βi.e
+`get('posts')`
2. Update your `getPosts.js` job to look like this:
+
+ Expand to see getPosts.js
+
```js title="getPosts.js"
// Get all posts
get('posts');
@@ -424,68 +292,39 @@ of how to set up `state.configuration` for `language-http`.
});
```
+
+
3. Now run the job using the following command
```bash
openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json
```
- And validate that you see the expected CLI logs:
+
+ And validate that you see the expected CLI logs:
```bash
[CLI] β Compiled job from getPosts.js
GET request succeeded with 200 β
[R/T] β Operation 1 complete in 120ms
[JOB] βΉ {
- userId: 1,
- id: 1,
- title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
- body: 'quia et suscipit\n' +
- 'suscipit recusandae consequuntur expedita et cum\n' +
- 'reprehenderit molestiae ut ut quas totam\n' +
- 'nostrum rerum est autem sunt rem eveniet architecto'
+ userId: 1,
+ id: 1,
+ title: 'sunt aut facere repellat provident occaecati excepturi optio reprehenderit',
+ body: 'quia et suscipit\n' +
+ 'suscipit recusandae consequuntur expedita et cum\n' +
+ 'reprehenderit molestiae ut ut quas totam\n' +
+ 'nostrum rerum est autem sunt rem eveniet architecto'
}
[R/T] β Operation 2 complete in 0ms
[CLI] β Writing output to tmp/output.json
[CLI] β Done in 470ms! β¨
- ```
-
-#### π Challenge: Fetch Covid-19 metadata
-1. Using the [disease.sh API](https://disease.sh/), write an operation that
- returns all covid-19 metadata.
-
-:::tip
-
-`https://disease.sh/v3/covid-19/` as your **baseUrl** in `state.configuration`
-
-:::
-
-2. Validate your output: there are a lot of ways you might choose to format or
- display this data. Share your results with your administrator for feedback.
-
-### 4. Additional arguments and commands
-
-#### π Challenge: Practice CLI arguments and commands
-
-Perform these tasks and submit answers to the discussion questions to your
-administrator for feedback.
-
-1. Compile a openfn job (**hello.js**).
-
- > What's the difference between the job you wrote and the compiled job?
-
-2. Run a job without "strict mode" enabled.
-
- > What's the difference between the outputs when strict mode is enabled and
- > disabled?
-
-3. Run a job with the log level set to `none`, and then run it again with the
- log level set to `debug`.
+ ```
- > When is it appropriate to use these different log levels?
+
-### 5. Manipulating data in a sequence of operations
+### 4. Clean & Transform Data
In most cases you need to manipulate, clean, or transform data at some step in
your workflow. For example after we get data from the
@@ -494,36 +333,31 @@ by user id. The example below shows how we can:
1. get all posts and return them in `state.data`
2. group returned posts by `userId`
-3. log posts with userId 1
+3. log posts with userId `1`
-##### Example:
+
+Expand to see example:
```js title="getPosts.js"
// Get all posts
get('posts');
-// Group posts by user id
-fn(state => {
- const posts = state.data;
+// Group posts by user id fn(state => { const posts = state.data;
- // Group posts by userId
- const groupPostsByUserId = posts.reduce((acc, post) => {
- const existingValue = acc[post.userId] || [];
- return { ...acc, [post.userId]: [...existingValue, post] };
- }, {});
+// Group posts by userId const groupPostsByUserId = posts.reduce((acc, post) =>
+{ const existingValue = acc[post.userId] || []; return { ...acc, [post.userId]:
+[...existingValue, post] }; }, {});
- // console.log(groupPostsByUserId);
- return { ...state, groupPostsByUserId };
-});
+// console.log(groupPostsByUserId); return { ...state, groupPostsByUserId }; });
-// Log posts where userId = 1
-fn(state => {
- const { groupPostsByUserId } = state;
- console.log('Post with userId 1', groupPostsByUserId[1]);
- return state;
+// Log posts where userId = 1 fn(state => { const { groupPostsByUserId } =
+state; console.log('Post with userId 1', groupPostsByUserId[1]); return state;
});
+
```
+
+
What is array.reduce?
The reduce() method applies a function against an accumulator and
@@ -535,17 +369,14 @@ the sum of all the elements in an array:
##### JavaScript Demo: `Array.reduce()`
```
+
const array1 = [1, 2, 3, 4];
-// 0 + 1 + 2 + 3 + 4
-const initialValue = 0;
-const sumWithInitial = array1.reduce(
- (accumulator, currentValue) => accumulator + currentValue,
- initialValue
-);
+// 0 + 1 + 2 + 3 + 4 const initialValue = 0; const sumWithInitial =
+array1.reduce( (accumulator, currentValue) => accumulator + currentValue,
+initialValue );
-console.log(sumWithInitial);
-// Expected output: 10
+console.log(sumWithInitial); // Expected output: 10
```
@@ -554,35 +385,23 @@ You can learn more about `array.reduce` from
-> Expected CLI logs
+>
-```
-[CLI] β Compiled job from getPosts.js
-GET request succeeded with 200 β
-[R/T] β Operation 1 complete in 825ms
-[R/T] β Operation 2 complete in 0ms
-[JOB] βΉ Post with userId 1 [
- //All of posts for userId 1
-]
-[R/T] β Operation 3 complete in 12ms
-[CLI] β Writing output to tmp/output.json
-[CLI] β Done in 1.239s! β¨
-```
+
+ Expand to see expected CLI logs
-#### π Challenge: extract names & emails
+```
-Using
-[https://jsonplaceholder.typicode.com/posts/1/comments](https://jsonplaceholder.typicode.com/posts/1/comments)
-API fetch comments for post with id 1 and extract name and email from each
-comment in that post
+[CLI] β Compiled job from getPosts.js GET request succeeded with 200 β [R/T] β
+Operation 1 complete in 825ms [R/T] β Operation 2 complete in 0ms [JOB] βΉ Post
+with userId 1 [ //All of posts for userId 1 ] [R/T] β Operation 3 complete in
+12ms [CLI] β Writing output to tmp/output.json [CLI] β Done in 1.239s! β¨
-1. Get post all comments for post id 1
-2. Extract name and email from comments
-3. Log the extracted data from comments
+```
-Discuss the results with your administrator.
+
-### 6. Debugging errors
+### 5. Debugging errors
When debugging, itβs interesting to use log to have a visual representation of
the content of the manipulated objects (such as state).
@@ -603,6 +422,10 @@ fn(state => {
##### Create **debug.js** and paste the code below
+
+
+ Expand to see debug.js
+
```jsx title="debug.js"
// Get all posts
get('posts');
@@ -616,22 +439,25 @@ fn(state => {
});
```
-##### Run **openfn debug.js -a http**
+
-> Expected CLI logs
+##### Run **openfn debug.js -a http**
-```bash
-[CLI] β TypeError: path.match is not a function
- at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26)
- at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22)
- at getPostbyIndex (vm:module(0):5:37)
- at vm:module(0):18:36
- at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12
- at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26
- at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
- at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18)
- at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20)
-```
+
+ Expected CLI logs
+ ```bash
+ [CLI] β TypeError: path.match is not a function
+ at dataPath (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:258:26)
+ at dataValue (/tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:262:22)
+ at getPostbyIndex (vm:module(0):5:37)
+ at vm:module(0):18:36
+ at /tmp/openfn/repo/node_modules/@openfn/language-common/dist/index.cjs:241:12
+ at file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:288:26
+ at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
+ at async run (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/node_modules/@openfn/runtime/dist/index.js:269:18)
+ at async executeHandler (file:///home/openfn/.asdf/installs/nodejs/18.12.0/lib/node_modules/@openfn/cli/dist/process/runner.js:388:20)
+ ```
+
As you can see from our logs that helper function `dataValue` has a TypeError,
To troubleshoot this you can go to the documentation for **dataValue ->
@@ -642,36 +468,25 @@ According to the docs, dataValue take path which is a string type. But in our
operation we were passing an integer, thatβs why we have a _TypeError_. You can
fix the error by passing a string in dataValue i.e `console.log(dataValue(β1β))`
-> Expected CLI logs
-
-```bash
-[CLI] β Compiled job from debug.js
-GET request succeeded with 200 β
-[R/T] β Operation 1 complete in 722ms
-[JOB] βΉ [Function (anonymous)]
-[R/T] β Operation 2 complete in 1ms
-[CLI] β Writing output to tmp/output.json
-[CLI] β Done in 1.102s β¨
-```
+
+ Expected CLI logs
+ ```bash
+ [CLI] β Compiled job from debug.js
+ GET request succeeded with 200 β
+ [R/T] β Operation 1 complete in 722ms
+ [JOB] βΉ [Function (anonymous)]
+ [R/T] β Operation 2 complete in 1ms
+ [CLI] β Writing output to tmp/output.json
+ [CLI] β Done in 1.102s β¨
+ ```
+
If you need more information for debugging you can pass -l debug which will give
all information about the run
i.e `openfn debug.js -a http -l debug`
-#### π Challenge: control error messages
-
-Debug what is causing an error on the following line of code and display the
-error message
-
-```jsx
-// Get post where id is 180
-get('posts/180');
-```
-
-Discuss the results with your administrator.
-
-### 7. Each and array iteration
+### 6. Each and array iteration
We often have to perform the same operation multiple times for items in an
array. Most of the helper functions for data manipulation are inherited from
@@ -679,6 +494,9 @@ array. Most of the helper functions for data manipulation are inherited from
##### Modify getPosts.js to group posts by user-ID
+
+Expand to see getPosts.js
+
```js title="getPosts.js"
// Get all posts
get('posts');
@@ -712,6 +530,8 @@ each('posts[*]', state => {
});
```
+
+
Notice how this code uses the `each` function, a helper function defined in
[language-common](/adaptors/packages/common-docs/#eachdatasource-operation--operation)
but accessed from this job that is using language-http. Most adaptors import and
@@ -719,35 +539,24 @@ export many functions from `language-common`.
##### Run **openfn getPosts.js -a http -o tmp/output.json**
-> Expected CLI logs
-
-```bash
-[CLI] β Compiled job from getPosts.js
-GET request succeeded with 200 β
-[R/T] β Operation 1 complete in 730ms
-[R/T] β Operation 2 complete in 0ms
-[R/T] β Operation 3 complete in 0ms
-[JOB] βΉ Posts [
-// Posts
-]
-[R/T] β Operation 4 complete in 10ms
-[CLI] β Writing output to tmp/output.json
-[CLI] β Done in 1.091s! β¨
-```
-
-#### π Challenge: Reduce, filter, and map
-
-Using Javascript globals i.e `Array.reduce`, `Array.filter` or `Array.map`,
-build function that will get posts by user id.
-
-1. Create a file called job1.js
-2. Add the 1st operation which is get all posts
-3. Add 2nd operation which has a function that filter posts by id
-4. Use the function from 2nd operation to get all post for user id 1
-
-Discuss the results with your administrator.
+
+ Expand to see expected CLI logs
+ ```bash
+ [CLI] β Compiled job from getPosts.js
+ GET request succeeded with 200 β
+ [R/T] β Operation 1 complete in 730ms
+ [R/T] β Operation 2 complete in 0ms
+ [R/T] β Operation 3 complete in 0ms
+ [JOB] βΉ Posts [
+ // Posts
+ ]
+ [R/T] β Operation 4 complete in 10ms
+ [CLI] β Writing output to tmp/output.json
+ [CLI] β Done in 1.091s! β¨
+ ```
+
-### 8. Running Workflows
+### 7. Running Workflows
As of `v0.0.35` the `@openfn/cli` supports running not only jobs, but also
_workflows_. Running a workflow allows you to define a list of jobs and rules
@@ -774,25 +583,29 @@ initial state.
A workflow is the execution plan for running several jobs in a sequence. It is
defined as a JSON object that consists of the following properties:
-- `start` (optional): The ID of the job that should be executed first (defaults
- to jobs[0]).
-- `jobs` (required): An array of job objects, each of which represents a
- specific task to be executed.
- - `id` (required): A job name that is unique to the workflow and helps you ID
- your job.
- - `configuration`: (optional) Specifies the configuration file associated with
- the job.
- - `data` (optional): A JSON object that contains the pre-populated data.
- - `adaptor` (required): Specifies the adaptor used for the job (version
- optional).
- - `expression` (required): Specifies the JavaScript file associated with the
- job. It can also be a string that contains a JavaScript function to be
- executed as the job.
- - `next` (optional): An object that specifies which jobs to call next. All
- edges returning true will run. The object should have one or more key-value
- pairs, where the key is the ID of the next job, and the value is a boolean
- expression that determines whether the next job should be executed.If there
- are no next edges, the workflow will end.
+```json
+{
+ "start": "a", // optionally specify the start node (defaults to jobs[0])
+ "jobs": [
+ {
+ "id": "a",
+ "expression": "fn((state) => state)", // code or a path
+ "adaptor": "@openfn/language-common@1.75", // specifiy the adaptor to use (version optional)
+ "data": {}, // optionally pre-populate the data object (this will be overriden by keys in in previous state)
+ "configuration": {}, // Use this to pass credentials
+ "next": {
+ // This object defines which jobs to call next
+ // All edges returning true will run
+ // If there are no next edges, the workflow will end
+ "b": true,
+ "c": {
+ "condition": "!state.error" // Note that this is an expression, not a function
+ }
+ }
+ }
+ ]
+}
+```
###### Example of a workflow
@@ -1043,8 +856,10 @@ outlined below:
```json
{
...
- "data": "tmp/initial-data.json",
+ "state": {
+ "data": "tmp/initial-data.json",
+ }
}
```
-:::
\ No newline at end of file
+:::
diff --git a/sidebars-main.js b/sidebars-main.js
index 6964393c081..e8064687a42 100644
--- a/sidebars-main.js
+++ b/sidebars-main.js
@@ -40,7 +40,8 @@ module.exports = {
items: [
'build-for-developers/cli-intro',
'build-for-developers/cli-usage',
- 'build-for-developers/cli-tutorial',
+ 'build-for-developers/cli-walkthrough',
+ 'build-for-developers/cli-challenges',
'build-for-developers/build-with-api',
'build-for-developers/security-for-devs',
//'build-for-developers/for-devs'
diff --git a/versioned_docs/version-legacy/cli.md b/versioned_docs/version-legacy/cli.md
index 270cdfef440..8ad7ee59874 100644
--- a/versioned_docs/version-legacy/cli.md
+++ b/versioned_docs/version-legacy/cli.md
@@ -535,12 +535,7 @@ administrator for feedback.
> What's the difference between the job you wrote and the compiled job?
-2. Run a job without "strict mode" enabled.
-
- > What's the difference between the outputs when strict mode is enabled and
- > disabled?
-
-3. Run a job with the log level set to `none`, and then run it again with the
+2. Run a job with the log level set to `none`, and then run it again with the
log level set to `debug`.
> When is it appropriate to use these different log levels?