-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into 209-get-google-sheets-data
- Loading branch information
Showing
13 changed files
with
346 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,6 +58,12 @@ This job is run *only once* as the Asana field gids for a given project are uniq | |
| Upon retrieving the data, OpenFn posts each individual Kobo survey data into the OpenFn inbox, to be processed by other jobs. | ||
| OpenFn.org and automatically triggers the next (third) job. | ||
|
|
||
| **Troubleshooting updates made in July 2024:** | ||
|
|
||
| 1. This job will not fetch Kobo submissions that are over 1 week old. If an undefined cursor or a cursor over 1 week old is used, the job will throw an error and ask the user to use a more recent cursor. This logic was implemented in July 2024 to prevent overwriting Asana tasks with Kobo data. | ||
| 2. This job logs the `_id` for each Kobo submission sent to the OpenFn Inbox. | ||
| 3. If for some reason an empty message is sent to the Inbox, the job will throw an error with the form id for further investigation. | ||
|
|
||
| ⚠ *Notes for developers:* | ||
| - An example of this `B. Fetch Kobo Grievance Data` job is linked to the Github file [`/asana/PullKoboGrievanceData.js`](https://github.com/OpenFn/ConSoSci/blob/master/asana/PullKoboGrievanceData.js). | ||
| - On OpenFn.org this job is configured with the `http` adaptor and a `cron` trigger. | ||
|
|
@@ -126,6 +132,59 @@ iv.Upsert the data into the Asana project, as follows: | |
| 3. The uuid used for syncing with the destination DB is the Kobo answer `_id`. The combination of `GrievanceID` and Kobo `_id_` creates a unique identifier for each form across various systems that would interact wit this data. **Note:** `uuid` may vary, and hence not a reliable unique identifier. | ||
|
|
||
|
|
||
| ## Q2 2024 GoogleSheets Integration | ||
|
|
||
| ### Project Overview | ||
|
|
||
| In some cases, after grievances are entered into the KoboToolbox form, the team will need to add updates on investigation and resolution in a Google Spreadsheet rather than in Asana. If they do that, we still need the data to be synced and updated with Asana. The objective of the project is to allow data entered in Kobo to be synced automatically in Google Sheet then manually updated by the team in Google Sheet, with all the updates being synced with Asana so that Asana contains the complete information on the grievance. | ||
|
|
||
|
|
||
| **GoogleSheets** | ||
|
|
||
| OpenFn will sync Kobo data to this [GoogleSheet](https://docs.google.com/spreadsheets/d/1WxZ8En1SX-g0UkvLnutZicmIEyZodwI4nGPFEvrxKE0/edit?gid=216423581#gid=216423581). Review the [GRM GoogleSheets User Guide](https://docs.google.com/document/d/1vAPLG1Sc4pSe6L0z3J5qVfmQFcvuJ1zEGmEKuExs5iI/edit) for details on how to use the GoogleSheet. | ||
|
|
||
|
|
||
|
|
||
| **Data Flows** | ||
|
|
||
| *[See this data flow diagram](https://lucid.app/lucidchart/3f6e91d6-feac-4c12-9602-60a0f9029943/edit?invitationId=inv_6fe97638-7d35-4930-9e00-ca2e538688eb&page=OQGlZYTqVO5E#).* | ||
|
|
||
|
|
||
| ### Jobs Configured | ||
|
|
||
| The following jobs are configured on OpenFn.org to run automatically. | ||
|
|
||
|
|
||
| **1. Sync to GoogleSheets** | ||
|
|
||
| After the tasks are upserted in Asana via the `GRM02. Upsert Aceh Grievances in Asana` job, the `Sync to GoogleSheets` job will run automatically. This job automatically cleans, maps, & loads the Kobo survey data into the specified GoogleSheet. This job stores the `Asana Task ID` returned from Asana in the Google sheet and uses it as the UUID for each row. | ||
| This job employs a *one-to-one mapping* i.e. | ||
|
|
||
| 1 Kobo form submission => 1 row in GoogleSheets | ||
|
|
||
| After OpenFn syncs the Kobo data to GoogleSheets, the Indonesia team addresses the grievances and leaves updates directly in the sheet. OpenFn has created protected ranges in the Sheet so that the users will only be able to update certain rows and cannot delete any rows. Refer to the [GRM GoogleSheets User Guide](https://docs.google.com/document/d/1vAPLG1Sc4pSe6L0z3J5qVfmQFcvuJ1zEGmEKuExs5iI/edit) for more details on these protected ranges. | ||
|
|
||
| **2. Update Asana Task** | ||
|
|
||
| This job is triggered by a message that is sent to the OpenFn project inbox. The message is automatically sent to OpenFn daily at midnight UTC by a Google Apps Script that was developed by the OpenFn team. Please notify the OpenFn team if any changes need to be made to this script. The message the script pushes to OpenFn will contain the rows and columns that have been updated since the last sync. Note: it is possible to send this message manually (instead of waiting until midnight) by clicking the "OpenFn Sync" button. Refer to the [GRM GoogleSheets User Guide](https://docs.google.com/document/d/1vAPLG1Sc4pSe6L0z3J5qVfmQFcvuJ1zEGmEKuExs5iI/edit) for more details. | ||
|
|
||
| The `Update Asana Task` job will find the existing task in Asana using the `Asana Task ID` and map and load the GoogleSheet data to Asana. Only the fields in the `MAP 2: GoogleSheets -> Asana` tab in the [mapping specifications](https://docs.google.com/spreadsheets/d/1D3_smWDjelubR_Lg-1xex9TLl6lAEGMSbGDyw8whqx4/edit#gid=373544466) will be synced from GoogleSheets to Asana. | ||
|
|
||
| ### Data Element Mappings | ||
|
|
||
| [See here](https://docs.google.com/spreadsheets/d/1D3_smWDjelubR_Lg-1xex9TLl6lAEGMSbGDyw8whqx4/edit#gid=373544466) for the integration mapping specifications. | ||
|
|
||
|
|
||
| ### Assumptions | ||
|
|
||
| 1. Only the GoogleSheets document owner and the WCS GoogleSheets integration user will be able to update the protected ranges in the GoogleSheet or delete rows in the Sheet. | ||
| 2. The `Update Asana Task` should always find the Asana task using the uuid `Asana Task Id`. If the task is not found in Asana it may have been deleted in Asana or someone may have changed the ID in the GoogleSheet. If assumption number 1 is met, only the WCS user and the document owner would have the privileges to update the ID the GoogleSheet. | ||
| 3. Because Asana tasks can be moved to different projects, Asana users should make sure OpenFn has access to those project spaces so that the integration will always find the task to be updated. | ||
| 4. The GoogleSheet sharing setting will remain set to "Restricted - Only people with access can open with the link" so that any changes made to the document will be associated with a user. | ||
|
|
||
|
|
||
|
|
||
|
|
||
| ### Administration & Support | ||
| #### Provisioning, Hosting, & Maintenance | ||
| This integration is hosted on OpenFn.org with hosted SaaS. The KoboToolBox Forms managed by WCS (email: [email protected]). | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| layout: page | ||
| title: Job Writing | ||
| nav_order: 6 | ||
| nav_order: 7 | ||
| permalink: /jobs/ | ||
| --- | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| layout: page | ||
| title: Kobo Form Sharing Notifications | ||
| nav_order: 6 | ||
| permalink: /form-sharing/ | ||
| --- | ||
|
|
||
| # Kobo Form Sharing Notification for WCS Admins | ||
|
|
||
| Lists of "deployed" and "archived" Kobo forms for data collection are stored in [this Google Sheet](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1559623602#gid=1559623602). | ||
|
|
||
| The below "Form Sharing" workflow has been configured in OpenFn. See the [form-sharing](https://github.com/OpenFn/ConSoSci/tree/master/form-sharing) directory for the underlying code. | ||
|
|
||
| When the workflow runs, it will: | ||
| 1. Check the connected Kobo accounts for form updates | ||
| 2. Compare any updated forms with the list of deployed forms in the ["Deployed" Forms Sheet](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1559623602#gid=1559623602) | ||
| 3. Add any newly deployed forms to the Sheet | ||
| 4. Update rows in the ["Deployed" Sheet](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1559623602#gid=1559623602) if forms are archived, and then add the archived form to the ["Archived" Sheet](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1965562058#gid=1965562058) | ||
| 5. Assign Asana Task(s) to the WCS admin to review every form newly deployed or archived | ||
|
|
||
|  | ||
|
|
||
| ## Specifications | ||
| - Original [Github technical specification](https://github.com/OpenFn/ConSoSci/issues/206) and the [workflow diagram (v2)](https://lucid.app/lucidchart/346b8e5c-6fb6-4a33-9d02-53e5059bd698/edit?invitationId=inv_d1431bce-05ae-4005-9b6a-9c279141a3a3&page=0_0#) | ||
| - [Change request](https://github.com/OpenFn/ConSoSci/issues/224) for how archived forms are managed |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| layout: page | ||
| title: Roadmap & Backlog | ||
| nav_order: 7 | ||
| nav_order: 8 | ||
| permalink: /roadmap/ | ||
| --- | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| layout: home | ||
| title: Training | ||
| nav_order: 7 | ||
| nav_order: 9 | ||
| permalink: /training/ | ||
| --- | ||
|
|
||
|
|
||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| //Check Kobo account for forms with these matching keywords | ||
| getForms({}, state => { | ||
| //ALL KEYWORDS: | ||
| //const keywords = ['price', 'prix', 'bns', 'nrgt', 'grm', 'feedback']; | ||
|
|
||
| //BNS KEYWORDS ONLY | ||
| const keywords = ['price', 'prix', 'bns', 'nrgt']; | ||
|
|
||
| const checkForKeyWords = name => { | ||
| return keywords.some(keyword => name.toLowerCase().includes(keyword)); | ||
| }; | ||
|
|
||
| state.activeForms = state.data.results | ||
| .filter(form => checkForKeyWords(form.name)) | ||
| .filter(form => form.deployment__active); | ||
|
|
||
| state.archivedForms = state.data.results | ||
| .filter(form => checkForKeyWords(form.name)) | ||
| .filter(form => !form.deployment__active); | ||
|
|
||
| state.data = {}; | ||
| state.references = []; | ||
| return state; | ||
| }); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| getValues( | ||
| '1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY', //googlesheet id | ||
| 'wcs-bns-DEPLOYED!A:O' //range of columns in sheet | ||
| ); | ||
|
|
||
| fn(state => { | ||
| const { activeForms, archivedForms, data } = state; | ||
| const [headers, ...sheetsData] = data.values; | ||
| const sheetsUids = sheetsData.map(row => row[0]); | ||
| console.log('Ignoring headers', headers); | ||
|
|
||
| state.formsToCreate = activeForms.filter( | ||
| form => !sheetsUids.includes(form.uid) | ||
| ); | ||
|
|
||
| state.formsToUpdate = archivedForms | ||
| .filter(form => sheetsUids.includes(form.uid)) | ||
| .map(form => { | ||
| const rowIndex = sheetsData.findIndex(row => { | ||
| return row[0] === form.uid; | ||
| }); | ||
| if (rowIndex !== -1) { | ||
| return { ...form, rowIndex }; | ||
| } | ||
| console.log(form.uid, 'Could not be found in google sheet'); | ||
| }); | ||
|
|
||
| return state; | ||
| }); | ||
|
|
||
| fn(state => { | ||
| const { data, references, response, ...remainingState } = state; | ||
|
|
||
| return remainingState; | ||
| }); |
Oops, something went wrong.