From ac36c879f31841e7e7548ef84cf9ce5781981a0c Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 13 Feb 2024 21:20:07 +0300 Subject: [PATCH 1/7] wip tutorial for http to googlesheets --- docs/tutorials/http-to-googlesheets.md | 143 +++++++++++++++++++++++++ docs/tutorials/tutorial.md | 10 -- sidebars-main.js | 2 +- 3 files changed, 144 insertions(+), 11 deletions(-) create mode 100644 docs/tutorials/http-to-googlesheets.md delete mode 100644 docs/tutorials/tutorial.md diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md new file mode 100644 index 00000000000..873a84c32b8 --- /dev/null +++ b/docs/tutorials/http-to-googlesheets.md @@ -0,0 +1,143 @@ +--- +sidebar_label: HTTP to GoogleSheets +title: HTTP to GoogleSheets Workflow +slug: /http-to-googlesheets +--- + +# Create a HTTP to GoogleSheets Workflow + +In this tutorial we are going to create a simple workflow that automate users +data sync between REST API and Google Sheets with OpenFn + +### Before you start + +Here are some things to know before you begin this process. + +- You have checked out our glossary and have an understanding of basic OpenFn + and API terminology. Check out the pages below to get started + - [OpenFn Concepts](../get-started/terminology.md) + - [A glossary for data integration](../get-started/glossary.md) +- You have a Google Account. We will use it to create GoogleSheets credentials +- You have a project on [app.openfn.org](app.openfn.org) + +## Getting started + +In this walkthrough, we will be setting up an **automatic data sync between HTTP +service and a GoogleSheet**. We will be syncing users data coming from a +JSONPlaceholder Rest API to a GoogleSheet. + +**This integration can be broken up into two parts:** + +1. Getting data from your source system to OpenFn input so you can inspect the + data structure to inform the job design for part two +2. Transforming and loading this data to your destination system + +… let’s get started! + +## Create a new Workflow + +To create a new Workflow in your Project: + +1. Go to the `project dashboard` page. +2. Click `Create new workflow` button. +3. Give your Workflow a descriptive `Name` (e.g., `Sync Users List`). +4. Choose your [Trigger](../build/triggers.md) +5. Edit your first [Step](../build/steps/steps.md) + +## Getting data from JSONPlaceholder + +[JSONPlaceholder](https://jsonplaceholder.typicode.com/users) provides a free +fake API for testing and prototyping. We will be using the +[Users Rest API](https://jsonplaceholder.typicode.com/users) for extracting +users data. This involves configuring a job in OpenFn to fetch users data via a +`GET` HTTP request. Click your first step to set up!, Configurate the step with +following options + +- Name `Fetch Users` +- Adaptor `Http` +- Version: `6.0.0` +- Credentials (Optional) - + `{ "baseUrl": "https://jsonplaceholder.typicode.com/"}` +- Step operations - `get("users")` If you have setup the credential above +- Input - `{}` + +**Once you are finished configuring and writing your job, save and run it!** + +**Check out the `Output & Log` panel to see if your run succeeded.** If it +succeeded, you should see: + +- Status `success` +- Log tab end with `Run complete with status: success` +- Input tab has `{}` +- Output tab has `{ data: [ {...}]}` + +## Transforming and loading usets data to Googlesheet + +1. **You should have a Googlesheet created with your google account email for + OpenFn to read and write data in your target GoogleSheet rows.** For this + demo, we have configured the Googlesheet + [like this](https://docs.google.com/spreadsheets/d/1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc/edit?usp=sharing) + to capture the users data. + +**Create a new step with the `googlesheet` adaptor for loading the users data +into your destination GoogleSheet.** + +- Credentials - GoogleSheet OAuth + - New Credentials + - GoogleSheet OAuth + - Login to your gmail + - Allow OpenFn + - Save your credential + +**Writing the job:** For this job we will use the `appendValues` operation to +add an array of rows to the spreadsheet. + +```js +// Users Data Model +fn(state => { + const users = state.data.map( + ({ id, name, username, address, phone, website, company }) => [ + id, + name, + username, + address.city, + phone, + website, + company.name, + ] + ); + + return { ...state, users }; +}); + +// Append User to Sheet +appendValues({ + spreadsheetId: '1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc', + range: 'users!A1:G1', + values: state => state.users, +}); +``` + +### Second Step - Sync Users + +- Name `Sync Users` +- Adaptor `googlesheet@latest` +- Credentials - GoogleSheet OAuth + - New Credentials + - GoogleSheet OAuth + - Login to your gmail + - Allow OpenFn + - Save your credential +- Job code - `// Users Data Model fn(state => { + + const users = state.data.map(({ id, name, username, address, phone, website, + company }) => [id, name, username, address.city, phone, website, + company.name]); + + return { ...state, users } }) + +// Append User to Sheet appendValues({ spreadsheetId: +'1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc', range: 'users!A1:G1', values: +state => state.users, })` + +- Input - `Fetch Users Final Output` diff --git a/docs/tutorials/tutorial.md b/docs/tutorials/tutorial.md deleted file mode 100644 index 8564566e0c8..00000000000 --- a/docs/tutorials/tutorial.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -title: Workflow Tutorial ---- -# Tutorial: Creating your first workflow - -:::warning Under construction - -This docs page is under construction. Check back later for the complete docs, or check out the Docs Version "Platform (v1)". - -::: diff --git a/sidebars-main.js b/sidebars-main.js index 856a6c12911..8f001b66394 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -28,7 +28,7 @@ module.exports = { { type: 'category', label: 'Tutorials', - items: ['tutorials/tutorial', 'tutorials/kobo-to-dhis2'], + items: ['tutorials/http-to-googlesheets', 'tutorials/kobo-to-dhis2'], }, { type: 'category', From 09e1c5abe642d3bfd92e8c910c502d0877ed3e38 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Tue, 13 Feb 2024 21:40:13 +0300 Subject: [PATCH 2/7] clean up --- docs/tutorials/http-to-googlesheets.md | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index 873a84c32b8..fd824c38125 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -117,27 +117,3 @@ appendValues({ values: state => state.users, }); ``` - -### Second Step - Sync Users - -- Name `Sync Users` -- Adaptor `googlesheet@latest` -- Credentials - GoogleSheet OAuth - - New Credentials - - GoogleSheet OAuth - - Login to your gmail - - Allow OpenFn - - Save your credential -- Job code - `// Users Data Model fn(state => { - - const users = state.data.map(({ id, name, username, address, phone, website, - company }) => [id, name, username, address.city, phone, website, - company.name]); - - return { ...state, users } }) - -// Append User to Sheet appendValues({ spreadsheetId: -'1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc', range: 'users!A1:G1', values: -state => state.users, })` - -- Input - `Fetch Users Final Output` From 757712a3ef1d31f82acf1d3296264e5402e188df Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 14 Feb 2024 11:27:31 +0300 Subject: [PATCH 3/7] improvements --- docs/tutorials/http-to-googlesheets.md | 113 ++++++++++++++----------- 1 file changed, 63 insertions(+), 50 deletions(-) diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index fd824c38125..b56cf9ff281 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -9,7 +9,7 @@ slug: /http-to-googlesheets In this tutorial we are going to create a simple workflow that automate users data sync between REST API and Google Sheets with OpenFn -### Before you start +## Before you start Here are some things to know before you begin this process. @@ -49,19 +49,19 @@ To create a new Workflow in your Project: [JSONPlaceholder](https://jsonplaceholder.typicode.com/users) provides a free fake API for testing and prototyping. We will be using the [Users Rest API](https://jsonplaceholder.typicode.com/users) for extracting -users data. This involves configuring a job in OpenFn to fetch users data via a +users data. This involves configuring a step in OpenFn to fetch users data via a `GET` HTTP request. Click your first step to set up!, Configurate the step with following options - Name `Fetch Users` -- Adaptor `Http` +- Adaptor `http` - Version: `6.0.0` - Credentials (Optional) - `{ "baseUrl": "https://jsonplaceholder.typicode.com/"}` - Step operations - `get("users")` If you have setup the credential above - Input - `{}` -**Once you are finished configuring and writing your job, save and run it!** +**Once you are finished configuring and writing your step, save and run it!** **Check out the `Output & Log` panel to see if your run succeeded.** If it succeeded, you should see: @@ -71,49 +71,62 @@ succeeded, you should see: - Input tab has `{}` - Output tab has `{ data: [ {...}]}` -## Transforming and loading usets data to Googlesheet - -1. **You should have a Googlesheet created with your google account email for - OpenFn to read and write data in your target GoogleSheet rows.** For this - demo, we have configured the Googlesheet - [like this](https://docs.google.com/spreadsheets/d/1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc/edit?usp=sharing) - to capture the users data. - -**Create a new step with the `googlesheet` adaptor for loading the users data -into your destination GoogleSheet.** - -- Credentials - GoogleSheet OAuth - - New Credentials - - GoogleSheet OAuth - - Login to your gmail - - Allow OpenFn - - Save your credential - -**Writing the job:** For this job we will use the `appendValues` operation to -add an array of rows to the spreadsheet. - -```js -// Users Data Model -fn(state => { - const users = state.data.map( - ({ id, name, username, address, phone, website, company }) => [ - id, - name, - username, - address.city, - phone, - website, - company.name, - ] - ); - - return { ...state, users }; -}); - -// Append User to Sheet -appendValues({ - spreadsheetId: '1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc', - range: 'users!A1:G1', - values: state => state.users, -}); -``` +## Transforming and loading users data to Googlesheet + +You should have a Googlesheet created with your google account email for OpenFn +to read and write data in your target GoogleSheet rows. For this demo, we have +configured the Googlesheet +[like this](https://docs.google.com/spreadsheets/d/1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc/edit?usp=sharing) +to capture the users data. + +Create a new step with the `googlesheet` adaptor for loading the users data into +your destination GoogleSheet. Configure the step with the following options + +- Name `Sync Users` +- Adaptor `googlesheetps` +- Version: `2.2.2` +- Credentials: Create new GoogleSheet OAuth credentials and save it +- Step operations: For this job we will use the `appendValues` operation to add + an array of rows to the spreadsheet. Make sure you update the `spreadsheetId` + to match the Id of your spreadsheet + + ```js + // Prepare array of users data + fn(state => { + const users = state.data.map( + ({ id, name, username, address, phone, website, company }) => [ + id, + name, + username, + address.city, + phone, + website, + company.name, + ] + ); + + return { ...state, users }; + }); + + // Append user data to GoogleSheet + appendValues({ + spreadsheetId: '1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc', + range: 'users!A1:G1', + values: state => state.users, + }); + ``` + +- Input - `Final output of Fetch Users` + +If you have already ran the `Fetch Users` step, you will have initial input to +test `Sync Users` step. Select the input from the input panel and click +`Create New Work Order` to run this step. + +## Time to test! + +1. Select and open inspector for `Fetch Users` step +2. Create a new empty input `{}` +3. Click `Create New Work Order` to execute the step +4. Check results in `Output & Logs` panel and ensure that both steps have passed + with status `success` +5. Finally check your spreadsheet to see the synced users data From 551ce8580f37790e1f9ce1804c2a2170b864ff89 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 14 Feb 2024 15:12:26 +0300 Subject: [PATCH 4/7] wip tutorials setup --- docs/tutorials/http-to-googlesheets.md | 1 - docs/tutorials/tutorial.md | 12 ++++++++++++ sidebars-main.js | 6 +++++- 3 files changed, 17 insertions(+), 2 deletions(-) create mode 100644 docs/tutorials/tutorial.md diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index b56cf9ff281..d7bf9d5c00d 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -1,7 +1,6 @@ --- sidebar_label: HTTP to GoogleSheets title: HTTP to GoogleSheets Workflow -slug: /http-to-googlesheets --- # Create a HTTP to GoogleSheets Workflow diff --git a/docs/tutorials/tutorial.md b/docs/tutorials/tutorial.md new file mode 100644 index 00000000000..91de74fc5d0 --- /dev/null +++ b/docs/tutorials/tutorial.md @@ -0,0 +1,12 @@ +--- +title: Workflow Tutorial +--- + +# Tutorial: Creating your first workflow + +:::warning Under construction + +This docs page is under construction. Check back later for the complete docs, or +check out the Docs Version "Platform (v1)". + +::: diff --git a/sidebars-main.js b/sidebars-main.js index 8f001b66394..86adc48e1f0 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -28,7 +28,11 @@ module.exports = { { type: 'category', label: 'Tutorials', - items: ['tutorials/http-to-googlesheets', 'tutorials/kobo-to-dhis2'], + items: [ + 'tutorials/tutorial', + 'tutorials/http-to-googlesheets', + 'tutorials/kobo-to-dhis2', + ], }, { type: 'category', From 13b1c69301701bb03cb6c04534f07b009ed78318 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 14 Feb 2024 16:29:55 +0300 Subject: [PATCH 5/7] fix broken link --- docs/tutorials/http-to-googlesheets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index d7bf9d5c00d..94fd4b3b70e 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -17,7 +17,7 @@ Here are some things to know before you begin this process. - [OpenFn Concepts](../get-started/terminology.md) - [A glossary for data integration](../get-started/glossary.md) - You have a Google Account. We will use it to create GoogleSheets credentials -- You have a project on [app.openfn.org](app.openfn.org) +- You have a project on [app.openfn.org](https://app.openfn.org) ## Getting started From d763935a9673ffccdef912879ef492785d224755 Mon Sep 17 00:00:00 2001 From: Emmanuel Evance Date: Wed, 14 Feb 2024 20:53:40 +0300 Subject: [PATCH 6/7] add video tutorial --- docs/tutorials/http-to-googlesheets.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index 94fd4b3b70e..a5cd240cd6a 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -129,3 +129,9 @@ test `Sync Users` step. Select the input from the input panel and click 4. Check results in `Output & Logs` panel and ensure that both steps have passed with status `success` 5. Finally check your spreadsheet to see the synced users data + +Check out the video overview below to learn how to create a HTTP to GoogleSheets +Workflow + + From db8e00068672aaa6c1f32a1408d700752bb1dec4 Mon Sep 17 00:00:00 2001 From: aleksa-krolls Date: Tue, 20 Feb 2024 23:34:54 +0200 Subject: [PATCH 7/7] edits to tutorial intro, headers, and new video embed link --- docs/tutorials/http-to-googlesheets.md | 105 ++++++++++++++++--------- 1 file changed, 66 insertions(+), 39 deletions(-) diff --git a/docs/tutorials/http-to-googlesheets.md b/docs/tutorials/http-to-googlesheets.md index a5cd240cd6a..206ba166f5e 100644 --- a/docs/tutorials/http-to-googlesheets.md +++ b/docs/tutorials/http-to-googlesheets.md @@ -3,37 +3,46 @@ sidebar_label: HTTP to GoogleSheets title: HTTP to GoogleSheets Workflow --- -# Create a HTTP to GoogleSheets Workflow +# Create a Workflow connecting a REST API to Google Sheets -In this tutorial we are going to create a simple workflow that automate users -data sync between REST API and Google Sheets with OpenFn +In this tutorial, we are going to walk through how to create a simple OpenFn +Workflow that automates syncing data between a REST API and Google Sheets, using +the `http` and `GoogleSheets` [Adaptors](/adaptors). + +## Video Walkthrough + +Watch the video and follow along with the steps below. + + ## Before you start -Here are some things to know before you begin this process. +Here are some we assume you've looked over before you begin this process. -- You have checked out our glossary and have an understanding of basic OpenFn - and API terminology. Check out the pages below to get started +- You have checked out our glossary and have an understanding of basic OpenFn & + API concepts. Check out the pages below to get started - [OpenFn Concepts](../get-started/terminology.md) - [A glossary for data integration](../get-started/glossary.md) -- You have a Google Account. We will use it to create GoogleSheets credentials -- You have a project on [app.openfn.org](https://app.openfn.org) +- You have a Google Account. We will use it to create a credential to authorize + with Google Sheets. +- You have access to an OpenFn project (either on a locally installed + [OpenFn v2 app](https://github.com/OpenFn/lightning) or on + [app.openfn.org](https://app.openfn.org)). ## Getting started -In this walkthrough, we will be setting up an **automatic data sync between HTTP -service and a GoogleSheet**. We will be syncing users data coming from a -JSONPlaceholder Rest API to a GoogleSheet. +In this walkthrough, we will configure a Workflow to **automatically sync `user` +data from a web REST API, and import this data to a GoogleSheet**. **This integration can be broken up into two parts:** -1. Getting data from your source system to OpenFn input so you can inspect the - data structure to inform the job design for part two -2. Transforming and loading this data to your destination system +1. Get data from the REST API (the "source" app) +2. Transforming & importing this data to a table in your GoogleSheet (the + "destination" app) -… let’s get started! +Let’s get started! -## Create a new Workflow +## 1: Create a new Workflow To create a new Workflow in your Project: @@ -43,7 +52,7 @@ To create a new Workflow in your Project: 4. Choose your [Trigger](../build/triggers.md) 5. Edit your first [Step](../build/steps/steps.md) -## Getting data from JSONPlaceholder +## 2. Configure your first Step to get data from the REST API [JSONPlaceholder](https://jsonplaceholder.typicode.com/users) provides a free fake API for testing and prototyping. We will be using the @@ -55,12 +64,23 @@ following options - Name `Fetch Users` - Adaptor `http` - Version: `6.0.0` -- Credentials (Optional) - +- Credentials (Optional: "Raw JSON" credential) - `{ "baseUrl": "https://jsonplaceholder.typicode.com/"}` -- Step operations - `get("users")` If you have setup the credential above -- Input - `{}` +- Job code: Add the operation `get("users")` in the code block if you've + configured the "Raw JSON" credential for the jsonplaceholder as the baseURL. + +:::tip Need help writing job code? + +Check out the docs on the ["http" Adaptor](/adaptors/packages/http-readme), +[configuring Steps](../build/steps/steps.md), and +[job-writing](../build/steps/jobs.md). -**Once you are finished configuring and writing your step, save and run it!** +::: + +**Once you are finished configuring and writing your Step, save and run it!** + +- See the [Workflows section](../build/workflows.md) for more guidance on + building & running Workflows. **Check out the `Output & Log` panel to see if your run succeeded.** If it succeeded, you should see: @@ -70,24 +90,26 @@ succeeded, you should see: - Input tab has `{}` - Output tab has `{ data: [ {...}]}` -## Transforming and loading users data to Googlesheet +## 3. Configure another Step to transform the data & import your GoogleSheet + +Create a new Googlesheet `Credential` using your Google account's email. (Make +sure this Google user has edit access to the GoogleSheet you want to integrate +with.) -You should have a Googlesheet created with your google account email for OpenFn -to read and write data in your target GoogleSheet rows. For this demo, we have -configured the Googlesheet +For this demo, we have configured the Googlesheet [like this](https://docs.google.com/spreadsheets/d/1gT4cpHSDQp8A_JIX_5lqTLTwV0xBo_u8u3ZNWALmCLc/edit?usp=sharing) -to capture the users data. +to store the `users` data. Create a new step with the `googlesheet` adaptor for loading the users data into your destination GoogleSheet. Configure the step with the following options - Name `Sync Users` -- Adaptor `googlesheetps` +- Adaptor `googlesheets` - Version: `2.2.2` -- Credentials: Create new GoogleSheet OAuth credentials and save it -- Step operations: For this job we will use the `appendValues` operation to add - an array of rows to the spreadsheet. Make sure you update the `spreadsheetId` - to match the Id of your spreadsheet +- Credentials: Create new `GoogleSheet OAuth` Credential and save it +- Step operations: For this job we will use the `appendValues()` operation to + add an array of rows to the spreadsheet. Make sure you update the + `spreadsheetId` to match the Id of your spreadsheet ```js // Prepare array of users data @@ -117,11 +139,11 @@ your destination GoogleSheet. Configure the step with the following options - Input - `Final output of Fetch Users` -If you have already ran the `Fetch Users` step, you will have initial input to -test `Sync Users` step. Select the input from the input panel and click -`Create New Work Order` to run this step. +If you have already ran the `Fetch Users` Step, you will have initial input to +test `Sync Users` Step. Select the input from the input panel and click +`Create New Work Order` to run this Step. -## Time to test! +## 4. Time to test! 1. Select and open inspector for `Fetch Users` step 2. Create a new empty input `{}` @@ -130,8 +152,13 @@ test `Sync Users` step. Select the input from the input panel and click with status `success` 5. Finally check your spreadsheet to see the synced users data -Check out the video overview below to learn how to create a HTTP to GoogleSheets -Workflow +Encountering errors or getting stuck? Check out the +[Workflow](../build/workflows.md) or +[Troubleshooting](../monitor-history/troubleshooting.md) docs. + +:::tip Are you blocked? Have questions? + +Reminder to [watch the video](#video-walkthrough) or post on the +[Community](https://community.openfn.org) to ask for help! - +:::