Allow users to upload inputs files

[hdoupe]

I'm getting ready to begin work on adding the capability for users to upload inputs files. From a high-level, it seems like users should be able to do three things:

1. Upload a single file for multiple major sections. For Tax-Brain, this would be a JSON file with a policy section and a behavior section.
2. Upload a file for each major section. For Tax-Brain, this would be a policy.json and a behavior.json file (or any other file extension like .csv)
3. Upload a file for a specific parameter. This would be helpful if the parameter is an array parameter that could potentially be very long or if it is an image parameter like a picture for some deep learning model.

One and two seem relatively straightforward, but 3 would require some work on the parameter processing side for either the model in functions.py or ParamTools. For this reason, I may begin with 1 and 2 and add support for 3 later.

I'll have more to say about the specifics for how this could work in the comments below.

cc @burkeob @Peter-Metz @jdebacker, you all may be interested in this.

[Peter-Metz]

@hdoupe this would be a great feature for Tax-Cruncher and could allow users to get results for multiple filing units at the same time. Looking forward to tracking your progress!

[hdoupe]

Status update: I've begun working on the file-reading capabilities on the lower-level IO packages ParamTools and compute-studio-storage:

• Read more file sources PSLmodels/ParamTools#111
• Add support for more storage object protocols compute-studio-storage#10

[hdoupe]

Some of the components are necessary:

• API endpoints for file uploads and downloads:

  • Upload: this should read in a file in chunks so that the web-server isn't over-loaded by a large file.

  • Download: this should probably also do downloads in chunks for large files. I was tempted to download directly from the storage bucket, but it's difficult to use our own authentication and data access logic on top of the cloud storage API unless the data is coming through the C/S webapp.

    Since this will go through the C/S API, we need some kind of token based authentication for granting access to the inputs files. Normally, this would just be the standard user authentication protocol, but model code will need to download the files, too. The model code shouldn't have access to the user's API token for getting it. This means that we will need a separate token that can be used to download the inputs while the simulation is running.

    I would love for this to fit in with fsspec's implementation framework now that ParamTools uses it as the primary file download interface.

• Frontend:

  • UI for uploading files and logic for chunk-uploading them to the server.
  • (Nice to have) UI for viewing uploaded files up to a certain size.