Add file upload api #267

dcmcand · 2021-01-26T15:38:39Z

Description of change
This PR adds the a API endpoint POST /files. This is only for the api, and does not include the front end.

How to test

Update .env file with new variables:
1. AWS_SECRET_ACCESS_KEY=EXAMPLEKEY
2. AWS_ACCESS_KEY_ID=EXAMPLEID
3. MINIO_ROOT_PASSWORD=EXAMPLKEY
4. MINIO_ROOT_USER=EXAMPLEID
5. S3_BUCKET=ttadp-test
6. If you aren't using docker, S3_ENDPOINT=http://localhost:9000, or if you are S3_ENDPOINT=http://minio:9000
Start app using yarn docker:start
Go to localhost:9000 to access the Minio web console (use AWS_SECRET_ACCESS_KEY for password and AWS_ACCESS_KEY_ID for username).
if you aren't using docker create a bucket named ttadp-test (It should be created automatically if you are using docker)
Open the web ui at http://localhost:3000 and save a Activity Report. Note the reportId to use in your curl command.
Open a terminal and type curl localhost:3000/api/files -F [email protected] -F reportId=1 -F attachmentType=ATTACHMENT where TESTFILE.txt is a file in the directory you run the curl command from
Output should be similar to

{
"id":5
}

Check the DB to ensure the appropriate metadata was added to the Files table

Issue(s)

part of API Endpoint for Document Upload #150 still needs front end integration
closes As a user, I want my uploaded files backed up (CP-9) #130
closes Selecting topics from the unfiltered list of all topics #235

Checklist

Meets issue criteria
Code tested
Meets accessibility standards (WCAG 2.1 Levels A, AA)
Documentation updated
- API methods
- Boundary and Data Flow Diagrams
- Architectural Decision Records written for major infrastructure decisions with the Nygard template
- OSCAL templates completed when security controls are implemented or modified

and FileStatus model

…nd' into add-file-upload-api

status if upload succeeds

src/lib/s3Uploader.js

dcmcand · 2021-01-26T15:42:51Z

src/lib/s3Uploader.js

+  }
+  return new Promise((resolve) => resolve(data));
+};
+


This should enable versioning on the bucket if it isn't enabled closing #130

Add actual topics

src/routes/files/handlers.js

rahearn

Great work! I'd like to update the File name to have the same casing structure as all other API fields, and I had some questions about a couple other things.

rahearn · 2021-01-26T18:55:52Z

docs/openapi/index.yaml

+        reportId: 
+          type: number
+          description: "id of the Activity report the file is associated with"
+        File:


I think for consistency, this should be named file

docker-compose.yml

rahearn · 2021-01-26T19:24:46Z

docs/openapi/paths/files.yaml

+          $ref: '../index.yaml#/components/schemas/fileUpload'
+  responses:
+    200:
+      description: One user by an id


I don't understand what this description means

That is cause it is nonsense. I think it got confused in a merge.

rahearn · 2021-01-26T19:26:39Z

src/lib/s3Uploader.js

+  }
+  return new Promise((resolve) => resolve(data));
+};
+


rahearn · 2021-01-26T19:31:54Z

src/lib/s3Uploader.js

+    Key: name,
+  };
+  // Only check for versioning if not using Minio
+  if (process.env.LOCAL_DEV !== 'true') {


Can we drop LOCAL_DEV in favor of process.env.NODE_ENV as it's being used here:

Head-Start-TTADP/src/app.js

Lines 71 to 75 in 4948bd3

if (process.env.NODE_ENV === 'production') {

app.use('*', (req, res) => {

res.sendFile(path.join(__dirname, 'client', 'index.html'));

});

}

Sounds like a good approach.

rahearn · 2021-01-26T20:14:11Z

src/routes/files/handlers.js

+      metadata = await createFileMetaData(originalFilename, fileName, reportId, attachmentType[0]);
+    } catch (err) {
+      await handleErrors(req, res, err, logContext);
+      res.status(500);


handleErrors already sets the status.

cool, that simplifies a bit.

rahearn · 2021-01-26T20:23:28Z

src/routes/files/handlers.test.js

+      });
+  });
+  it('checks the metadata was uploaded to the database', async () => {
+    const file = await File.findAll({ where: { id: fileId } });


This could be File.findOne and then all of the file[0] could be replaced with file

rahearn · 2021-01-26T20:27:53Z

src/routes/files/handlers.js

+  };
+  let file;
+  try {
+    await sequelize.transaction(async (transaction) => {


What's the benefit of wrapping this (and the update call in updateStatus) in a transaction?

I don't know, I am new to sequalize and found that was how josh had done it elsewhere, so I copied it. I THINK it would be fine not too.

rahearn · 2021-01-26T20:50:52Z

src/lib/s3Uploader.js

+export const s3 = new S3({
+  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
+  endpoint: process.env.S3_ENDPOINT,
+  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,


I have another requested change for this as well. In deployed environments, we should pull these variables (as well as the bucket name) out of the process.env.VCAP_SERVICES environment variable.

Drop attachment enum type

Update deployed env vars

…Head-Start-TTADP into cm-pr-267-add-file-upload-api

Co-authored-by: Ryan Ahearn <[email protected]>

…Head-Start-TTADP into cm-pr-267-add-file-upload-api

update file upload api

rahearn

commit d1c5786 Merge: 44eeeb8 059425c Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 17:10:20 2021 -0500 Merge pull request #157 from HHS/user-last-login Allow filtering users by last login and access permissions commit 059425c Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 15:25:09 2021 -0500 Make eslint happy commit 09e906f Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 14:46:50 2021 -0500 Refactor user filtering to match access control SOP rules directly commit facfee4 Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 14:03:29 2021 -0500 Allow filtering by only showing users who do have SITE_ACCESS commit b5b93f0 Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 13:40:44 2021 -0500 Allow filtering by last login > 60 or 180 days ago Options to match process here: https://github.com/HHS/Head-Start-TTADP/wiki/Access-Control-&-Account-Management-SOP#account-review-frequency-and-process commit 1d08788 Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 12:18:39 2021 -0500 Display lastLogin on admin user details commit 31883dd Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 11:12:00 2021 -0500 Add lastLogin to user api response commit 60bbefd Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 10:20:03 2021 -0500 Add lastLogin value to user model & update on login commit 1a2dd2c Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 09:36:09 2021 -0500 Prevent validation issues if HSES email updates commit 7554928 Merge: 875f5e2 44eeeb8 Author: Ryan Ahearn <[email protected]> Date: Tue Feb 9 14:15:51 2021 -0500 Merge pull request #286 from adhocteam/main Fix cucumber test on CI commit 875f5e2 Merge: 4bd8566 9f5bb95 Author: Ryan Ahearn <[email protected]> Date: Fri Feb 5 16:31:48 2021 -0500 Merge pull request #284 from adhocteam/main Filter locked users from admin list commit 4bd8566 Merge: 17f9d7f f327246 Author: Ryan Ahearn <[email protected]> Date: Fri Feb 5 16:05:29 2021 -0500 Merge pull request #281 from adhocteam/main Case insensitive admin search, testing updates and manager setting of notes and report status commit 17f9d7f Merge: 16d54e2 8e5d3ef Author: Ryan Ahearn <[email protected]> Date: Wed Feb 3 16:09:21 2021 -0500 Merge pull request #279 from adhocteam/main add frontend for file upload commit 16d54e2 Merge: 541640a e27e7df Author: Ryan Ahearn <[email protected]> Date: Tue Feb 2 15:33:25 2021 -0500 Merge pull request #275 from adhocteam/main Add goal component to frontend commit 541640a Merge: 790ca05 4c3a436 Author: Ryan Ahearn <[email protected]> Date: Fri Jan 29 15:47:13 2021 -0500 Merge pull request #272 from adhocteam/main Save collaborators to report commit 790ca05 Merge: 4948bd3 3cff2f4 Author: Ryan Ahearn <[email protected]> Date: Wed Jan 27 12:04:04 2021 -0500 Merge pull request #267 from adhocteam/main Add file upload api

…ctivity-reports Add ability to delete a report in My Alerts

Chuck McAndrew added 30 commits January 13, 2021 14:13

Add minio to docker-compose for local dev

703f4fc

Add handler for file upload endpoint

a41b773

add POST /files endpoint to upload files to s3

6881b4e

fmt

d8cdb09

fmt

788d8ef

Add model, migrations and seeders for File

ed89a9d

and FileStatus model

Add db logic to handler.js for file upload

237e764

Move s3Uploader to lib/s3Uploader.js

cb0e6bf

update branch with latest from main

8e30cac

Add first test

d45bbd4

Add versioning tests

6a4050a

Add s3 Uploader test

d0b267c

update .env.example

58ff705

remove debugging statement

8378bc4

Update files model

61d8165

Merge remote-tracking branch 'origin/js-saving-activity-report-fronte…

630680b

…nd' into add-file-upload-api

update create-files migration

d2cff7f

update files/handlers.js to update

9b71eaf

status if upload succeeds

merge in main

386086f

merge main

f8ca042

merge main

9a22bd3

fix merge

e0ae000

Merge branch 'main' into add-file-upload-api

61b57f0

merge main

59be6bb

merge main

345db1a

merge main

e27bb8b

merge main

8774025

update .env.example for non-docker development

168b6d8

update deps

fea7f25

fmt

9e9b0b4

rahearn self-requested a review January 26, 2021 15:39

dcmcand commented Jan 26, 2021

View reviewed changes

src/lib/s3Uploader.js Show resolved Hide resolved

dcmcand commented Jan 26, 2021

View reviewed changes

Merge pull request #129 from adhocteam/js-235-add-topics

58a854f

Add actual topics

dcmcand commented Jan 26, 2021

View reviewed changes

src/routes/files/handlers.js Show resolved Hide resolved

rahearn added 3 commits January 26, 2021 11:42

Merge branch 'main' into update-deployed-env-vars

aa912bc

Add user management workflow diagrams

eb70469

Move rds & s3 inside of accreditation boundary

0551a81

rahearn requested changes Jan 26, 2021

View reviewed changes

rahearn reviewed Jan 26, 2021

View reviewed changes

jasalisbury and others added 3 commits January 26, 2021 15:15

The file attachment migration drops the file enum type

4eeb54a

Set sandbox branch

aab85eb

update file upload api

2c03fe8

This was referenced Jan 26, 2021

Update file upload api #270

Closed

update file upload api adhocteam/Head-Start-TTADP#132

Merged

jasalisbury and others added 11 commits January 26, 2021 15:42

Merge pull request #131 from adhocteam/js-drop-attachment-enum

570207e

Drop attachment enum type

Merge branch 'main' into update-deployed-env-vars

8b070ef

Merge pull request #128 from HHS/update-deployed-env-vars

89ec60d

Update deployed env vars

Cleanup s3 variables

1232a3f

update file upload api

1e009d2

Cleanup s3 variables

3a38aa3

Merge branch 'cm-pr-267-add-file-upload-api' of github.com:adhocteam/…

99005ad

…Head-Start-TTADP into cm-pr-267-add-file-upload-api

Update src/lib/s3Uploader.js

04bbf74

Co-authored-by: Ryan Ahearn <[email protected]>

update s3uploader.js

9618281

Merge branch 'cm-pr-267-add-file-upload-api' of github.com:adhocteam/…

17ea941

…Head-Start-TTADP into cm-pr-267-add-file-upload-api

Merge pull request #132 from adhocteam/cm-pr-267-add-file-upload-api

3cff2f4

update file upload api

rahearn approved these changes Jan 27, 2021

View reviewed changes

rahearn merged commit 790ca05 into HHS:main Jan 27, 2021

rahearn pushed a commit that referenced this pull request May 10, 2021

Merge pull request #267 from adhocteam/TTAHUB-83-enable-deleting-of-a…

05305bb

…ctivity-reports Add ability to delete a report in My Alerts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add file upload api #267

Add file upload api #267

dcmcand commented Jan 26, 2021 •

edited

Loading

dcmcand Jan 26, 2021

rahearn Jan 26, 2021

rahearn left a comment

rahearn Jan 26, 2021

rahearn Jan 26, 2021

dcmcand Jan 26, 2021

rahearn Jan 26, 2021

rahearn Jan 26, 2021

dcmcand Jan 26, 2021

rahearn Jan 26, 2021

dcmcand Jan 26, 2021

rahearn Jan 26, 2021

rahearn Jan 26, 2021

dcmcand Jan 26, 2021

rahearn Jan 26, 2021

rahearn left a comment

	if (process.env.NODE_ENV === 'production') {
	app.use('*', (req, res) => {
	res.sendFile(path.join(__dirname, 'client', 'index.html'));
	});
	}

Add file upload api #267

Add file upload api #267

Conversation

dcmcand commented Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahearn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahearn left a comment

Choose a reason for hiding this comment

dcmcand commented Jan 26, 2021 •

edited

Loading