Pull grant and grantee data from HSES on a schedule #248
Conversation
…into kw-pull-hs-data
Doc: Add cf login to DB instructions
…tart-TTADP into kw-pull-hs-data
Pull grant and grantee data from HSES
There was a problem hiding this comment.
Hi @kryswisnaskas, good stuff. My biggest concern is the mixing of explicit IDs with auto-increment columns. As we saw with the user seeders in dev and sandbox doing that will likely result in duplicate IDs where we didn't mean to have them.
src/app.js
Outdated
| const timezone = 'America/New_York'; | ||
|
|
||
| // tmp schedule for testing | ||
| const schedule = '*/25 * * * *'; |
There was a problem hiding this comment.
Please update to the real schedule. I'm also curious how this will work in production. Production will have multiple instances running (2 at first), but we want to make sure that this is only running on one of them.
See the last section on CF_INSTANCE_INDEX of cloud.gov's docs for information on limiting to the first instance.
There was a problem hiding this comment.
Ah, yes, this slipped by. I'll take a look at the link. Thanks.
tools/importPlanGoals.test.js
Outdated
| }); | ||
|
|
||
| it('should import or update grants', async () => { | ||
| const grantsBefore = await Grantee.findAll(); |
There was a problem hiding this comment.
I think you meant to check the count of Grant before the test?
| const grantsBefore = await Grantee.findAll(); | |
| const grantsBefore = await Grant.findAll(); |
…tart-TTADP into kw-pull-hs-data
Update README and .env.example to include documentation about setting `AUTH_CLIENT_ID` and `AUTH_CLIENT_SECRET` variables.
Refactor test, auto increment, adjust goals import.
Update README and .env.example
.circleci/config.yml
Outdated
| hses_data_file_url: HSES_DATA_FILE_URL | ||
| hses_data_username: HSES_DATA_USERNAME | ||
| hses_data_password: HSES_DATA_PASSWORD |
There was a problem hiding this comment.
Wish I had caught this earlier. We have one endpoint for dev/sandbox/staging, and another for production, so the prod variables need to be different than what we use for staging.
| hses_data_file_url: HSES_DATA_FILE_URL | |
| hses_data_username: HSES_DATA_USERNAME | |
| hses_data_password: HSES_DATA_PASSWORD | |
| hses_data_file_url: PROD_HSES_DATA_FILE_URL | |
| hses_data_username: PROD_HSES_DATA_USERNAME | |
| hses_data_password: PROD_HSES_DATA_PASSWORD |
There was a problem hiding this comment.
Ah, good catch. Cut and paste omission.
tools/importPlanGoals.js
Outdated
| grantId = dbGrant.id; | ||
| currentGranteeId = dbGrant.granteeId; | ||
| // } |
Description of change
Adding a cron job, which is part of the server and runs everyday at midnights.
The job downloads data from HSES (in production env), extracts the zip archive into "temp" directory. Parses the xml data and populates or updates the SmartHub db with downloaded data.
How to test
Make sure the tests pass.
The imported data can be seen on sandbox in the database using the following:
(bonus test) additionally, to verify the download:
Issue(s)
Checklist