Provide data/form guards for common cloud storage providers out-of-the-box

[SergioBenitez]

A common use-case is streaming data from a client to file storage in the cloud. For instance, a file upload that is destined to be stored in Amazon S3. To prevent users from needlessly writing to disk, the TempFile guard does not implement AsyncRead: if the disk is not the final destination, it is likely the wrong tool. This forces the user to rethink their strategy (#1646, #1661), a good thing, at the expense of needing to write a custom data or form guard to accomplish their ultimate task.

We should remove this expense by proving data guards (i.e, S3File, AzureFile, etc) that stream directly to common cloud storage. To do so, we'd need to answer the following questions:

• Where would this functionality live?
• Which HTTP client would we use?
• How would timeouts and limits for outbound requests/responses be configured?
• How would we handle credentials?
• Besides Amazon/Azure/Google, are there other big providers we should support?
• Besides an object identifier, what other information should the guards expose?

[stvcooke]

What about similar entities for file storage but aren't full fledged cloud providers, such as Google Drive, Dropbox, Microsoft Sharepoint, MEGA?

[jebrosen]

What about similar entities for file storage but aren't full fledged cloud providers, such as Google Drive, Dropbox, Microsoft Sharepoint, MEGA?

There is no specific reason they couldn't be supported, but the overall feature set and security models are pretty different from backend cloud storage. For instance, there is a big difference between uploading to a single account known in advance vs authenticating users to upload files to their accounts - both in what is likely to be considered acceptable use of these services, and in the technical requirements on the implementation.

[timando]

I tried to implement uploading directly to S3, but I ran into a lifetime issue. The Data struct has a non-'static lifetime, but all the HTTP client libraries I could find require the body to be 'static.

[strowbeary]

I think a lot of back end cloud storage providers follow the openstack object storage standard (Similar to AWS S3).

[kotx]

Hi, I might have a use case for this.
I want to hash a file as it's streaming in, then persist it somewhere (be it local FS, S3, etc).
Currently, is there an easy way to do this for local FS from a form/data guard without reimplementing TempFile? I'd like to avoid reading the file back from disk if possible.