diff --git a/docs/2-bns-nrgt.md b/docs/2-bns-nrgt.md index 9b33abc..a0db97b 100644 --- a/docs/2-bns-nrgt.md +++ b/docs/2-bns-nrgt.md @@ -14,11 +14,6 @@ permalink: /bns_nrgt/ - [x] NRGT Historical --> OpenFn job [`nrgt/2017.js`](https://github.com/OpenFn/wcs/blob/master/ngrt/2017.js) - [x] NRGT Current --> OpenFn job [`nrgt/2019.js`](https://github.com/OpenFn/wcs/blob/master/ngrt/2019.js) -## OpenFn jobs to _fetch_ Kobo data - -- [x] Ongoing, timer-based fetch --> OpenFn job [`getKoboData.js`](https://github.com/OpenFn/wcs/blob/master/bns/getKoboData.js) -- [x] Historical, once-off fetch --> OpenFn job [`historical.js`](https://github.com/OpenFn/wcs/blob/master/bns/historical.js) - ## Mappings [See here](https://docs.google.com/spreadsheets/d/1EuSCOepC3gs8nRHlh9E4Tszi5txv__WxHkRAK80FMT4/edit#gid=0) for the mapping specifications defined by WCS. @@ -31,29 +26,19 @@ permalink: /bns_nrgt/ 1. For idempotency, the jobs create a new Kobo `uuid` to map to the DB `DatasetUuidId`. This is a concatenation of Kobo `_id` + `_submission_time` + `_xform_id_string`. We cannot use Kobo `_uuid` because this is refreshed every time a Kobo submission is cleaned. 2. All data cleaning will be done in Kobo Toolbox. Every time Kobo data is synced with the DB, it will overwrite the records saved there and use the above `uuid` to upsert existing records. 3. In the [`bns/survey.js`](https://github.com/OpenFn/wcs/blob/master/bns/survey.js) job, we utiilize some of the Kobo form metadata to create data for the `bns_matrix` [L52-L65](https://github.com/OpenFn/wcs/blob/master/bns/survey.js#L52-L65) and `nr` [L42-L50](https://github.com/OpenFn/wcs/blob/master/bns/survey.js#L42-L50) question groups. It is therefore important that future versions of this form follow the same Kobo question naming conventions, otherwise the data will _not_ map as expected and the job may need to be modified. -4. OpenFn will only automatically fetch the Kobo surveys where the form Ids have been specified in the job [`getKoboData.js`](https://github.com/OpenFn/wcs/blob/master/bns/getKoboData.js). This is to allow WCS to identify which forms have historical data to be migrated one time using [`historical.js`](https://github.com/OpenFn/wcs/blob/master/bns/historical.js), and which forms should be synced on an ongoing basis using [`getKoboData.js`](https://github.com/OpenFn/wcs/blob/master/bns/getKoboData.js). +4. OpenFn will only automatically fetch the Kobo surveys where the form Ids have been specified in the [Google Sheet Deployed Forms List](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1965562058#gid=1965562058). -## Data Flows Supported +## Syncing Kobo Form Submissions and the Workflows Supported ### (1) Scheduled Data Integration (Ongoing Sync) -1. On a scheduled-basis (e.g., every 3 hours), the OpenFn job [`1A. Get Kobo Data`](https://www.openfn.org/projects/1168/jobs/3562) (aka `getKoboData.js`) will run to fetch Kobo form data in bulk for the specified form Ids. _Before running the job, WCS should..._ - -- 1a. Update the survey Ids to fetched from Kobo toolbox (these can be copied from the URL of a Kobo form). In `https://kf.kobotoolbox.org/#/forms/aopf2bJ4cVqEXCrjnwAoHd/landing` then the string `aopf2bJ4cVqEXCrjnwAoHd` is the survey Id. -- 1b. Add the appropriate survey tag to indicate which mappings should be used to process the data. Tag options include: `bns_survey`, `bns_price`, `nrgt_current`, `nrgt_historical`. +1. On a scheduled-basis (e.g., every 3 hours), a OpenFn job will run to fetch Kobo form data in bulk for the specified form Ids. See the [Form Sharing docs](./form-sharing/#syncing-bnsnrgt-kobo-form-submissions) for more on how this workflow works and can be managed. -``` -surveys: [ - //** Specify new forms to fetch here **// - //** Tag options: bns_survey, bns_price, nrgt_current, nrgt_historical **// - { id: 'aMpW7wwRBRbtCfidtK2rRn', tag: 'bns_survey' }, //BNS Ndoki 2019 - { id: 'new-form-id', tag: 'form_tag' }, //New Form Name -``` +Note that this workflow will: +- Post each individual Kobo survey back to the OpenFn inbox as an individual Message. +- Message filter triggers will execute the relevant jobs (see above list) to process & load the data into the connect DB. -2. This job will post each individual Kobo survey back to the OpenFn inbox as an individual Message. -3. Message filter triggers will execute the relevant jobs (see above list) to process & load the data into the connect DB. -4. View **Activity History** to monitor the success of these integration flows. -5. If any Kobo data is cleaned, it will be fetched in the next job run (see step #1) and will overwrite\* the matching record in the DB. +View **Activity History** to monitor the success of these integration flows. If any Kobo data is cleaned, it will be fetched in the next job run and will overwrite\* the matching record in the DB. \*Note these jobs have built-in transformations to create a **custom unique identifier** to map to the DB column `DatasetUuidId`, which can be used in the OpenFn job `upsert()` operations to ensure idempotency. @@ -62,16 +47,7 @@ surveys: [ ``` ### (2) Historical Kobo Migrations (Once-off) - -**Note:** OpenFn jobs support historical migrations for a total of 5,000 Kobo submissions. If WCS wishes to migrate a list of forms where the total submissions exceed 5k, it should either (a) break up the migration and repeat the following steps for smaller subsets of the forms, or (b) contact OpenFn for support running larger bulk migrations locally. - -1. At any time, the OpenFn job [`1B. Get Historical Kobo Data`](https://www.openfn.org/projects/1168/jobs/3542) (aka `historical.js`) can be run on-demand to manually fetch historical Kobo data in bulk. _Before running the job, WCS should..._ - -- 1a. Update the survey Ids to fetched from Kobo toolbox (these can be copied from the URL of a Kobo form). In `https://kf.kobotoolbox.org/#/forms/aopf2bJ4cVqEXCrjnwAoHd/landing` then the string `aopf2bJ4cVqEXCrjnwAoHd` is the survey Id. -- 1b. Add the appropriate survey tag to indicate which mappings should be used to process the data. Tag options include: `bns_survey`, `bns_price`, `nrgt_current`, `nrgt_historical`. - -2. When ready to sync the historical data, click "Run job" button. - ![run-job](../run-this-job.png) +If desired, Kobo forms submissions can be "re-synced" for historical or archived Kobo forms. See the [Form Sharing docs](./form-sharing/#syncing-bnsnrgt-kobo-form-submissions) for more on how this workflow works and can be managed. ### (3) Real-Time Integration (Not used, available as needed) @@ -91,14 +67,8 @@ If WCS would like to integrate a new BNS or NRGT Kobo form with the database... 1. Delete any test submissions used in training/ testing. 2. Check if the form already has real submission data collected. If yes, add the `formId` to the [`1B. Get Historical Kobo Data`](https://www.openfn.org/projects/1168/jobs/3542) job and run it to first migrate the historical data to the database (see more in data flow #2 above). -3. Add the new `formId` and relevant `tag` to the job [`1A. Get Kobo Data`](https://www.openfn.org/projects/1168/jobs/3562) to fetch data for this form on an ongoing basis. +3. Add the new `formId` and relevant info to the [Google Sheet Deployed Forms List](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1965562058#gid=1965562058) to fetch data for this form on an ongoing basis. (See the [Form Sharing docs](./form-sharing) for more on how this process.) ### Disconnecting Kobo Forms -To remove a Kobo form from the integration flow, edit the job [`1A. Get Kobo Data`](https://www.openfn.org/projects/1168/jobs/3562) (aka `getKoboData.js`) to remove the relevant form Id. - -## Open Questions - -1. Can WCS test to confirm the jobs are mapping data to Mssql as expected? [See this video for guidance](http://somup.com/cYQjQxX02A). Please pay special attention to the BNS Survey job, where the mapping logic is more complex and based on the form metadata. -2. Can WCS confirm the Postgres database we should also map these to? And should we map the exact same data to Postgres as we are to Mssql? Or will different surveys be mapped to different DBs? - +To remove a Kobo form from the integration flow, remove it from the [Google Sheet Deployed Forms List](https://docs.google.com/spreadsheets/d/1s7K3kxzm5AlpwiALattyc7D9_aIyqWmo2ubcQIUlqlY/edit?gid=1965562058#gid=1965562058) or mark the column `automate_sync: false`.