break: secure file upload and download (public and private)

[gunar]

Card: https://trello.com/c/cS7bfBXL/2019-restrict-access-to-user-uploaded-files-to-bops-uniform-only

• prevents user uploads from being publicly accesible
• makes all S3 routes go through the API
• adds a new API Token concept to only allow BoPS to download user files
• side-effect: prevents users from downloading their own files

[github-actions]

Removed vultr server and associated DNS entries

[gunar]

This is now ready for review. Sorry for being trigger-happy with the review request earlier (ie asking review before tests were passing).

[gunar]

Hey @jessicamcinchak, quick question. This PR requires adding a new env var. I've added it to s3://pizza-secrets/api.env.test which makes the CI tests pass. Now as for the Pizza, what would be the right place to put this API_FILE_KEY environment variable? Here maybe?

planx-new/.github/workflows/pull-request-main.yml

Line 347 in 3d0b63f

ROOT_DOMAIN=${{ env.FULL_DOMAIN }}

[jessicamcinchak]

hey @gunar - without fully reviewing this PR yet, I'd suggest adding the new variable to the root .env file as well as the API's env.test and then it'll be available in the pizza's docker container and during local development.

any variables defined like L347 X=${{ env.X }} will be printed by Actions fyi; Github only masks values like X=${{ secrets.X }} (we don't care about this for the domain here as it's not sensitive, but we would for the new file key obviously!). I'm generally aiming to decrease how many variables we need to keep in Github Secrets and use remote root.env file as source of truth for all environments using docker 🙂

lemme know if you have any issues with this approach though!

[gunar]

That makes sense, and it worked! Thanks!

[jessicamcinchak]

thanks for all your work & patience on this one, apologies it's been a particularly slow review process as this one just touches so much & requires more cross-project communication than usual!

i'm happy with the code & went through a full test LDC service: file upload components, review page, and CSV all look good - but both submissions to Uniform & BOPS failed (you can see logs here). Once file conflicts are resolved here, I'm happy to try again - I think once we have a successful staging submission to both BOPS & Uniorm and can confirm that they are able to pick up the files as expected then this could merge 👍

[gunar]

Next steps:

1. Rebase
2. Submit application from Pizza to BOPS Staging
3. Check that it was submitted correctly to Uniform
4. Give BOPS token, ask them to add it to the their request's headers, confirm it worked
5. Merge 🎉

[gunar]

Thank you for the review @jessicamcinchak—all good points. I believe I've addressed them. I've added the thing about how to rotate randomPassword as an in-place comment—given that we're only using it in that single place for now.

[gunar]

Pizza isn't building. I'll investigate.

[gunar]

It's Friday I'm in… a ready-for-review state 😂 🎵

[gunar]

Worth noting that I had to change this otherwise tsc would try to compile the whole sdk and the GH Action worker would run out of memory.

- import { S3 } from "aws-sdk";
+ import S3 from "aws-sdk/clients/s3";

In a separate PR I'll suggest increasing the maximum amount of memory available for the worker (just in case).

[gunar]

Worth noting about the "worth noting" above that the Docker build would fail silently (ie the error message was there but it would return a Linux zero code) which made it hard for me to find the root cause at first.

[gunar]

Actually, we already had a plan for this PR didn't we? Sorry I forgot.

@jessicamcinchak on Tuesday when you're back do you mind submitting an application from this Pizza into BOPS Staging? Happy to learn how to do it myself too but at this point I just want this PR to be over with 😅

[gunar]

Restrict access to user-uploaded files to BOPS/Uniform only