Use Case for streaming files (upload & download)

[emedina]

I would like to discuss the following use case with you to get your feedback and best practices.

Let's say that I'm working with a list of documents (or any other binary content) that is stored in some storage (S3, NFS, etc.). Then, the user can request that list of documents, upload new ones or download an existing one. Let's focus on the upload/download for now.

One approach that I've been using when implementing the typical 3-layered architecture (Web, Business, Data) is to either work with a file (File type in Java) or the input stream (InputStream type in Java).

Using a file means that I could get the document from the user in the web layer (as multipart/form-data, for instance) and save it directly into a temporary file. This approach would allow me to avoid propagating the input stream across all the layers until the one that eventually saves it to the provided storage, hence avoiding the usage of "infra/technical" concerns (InputStream type in Java) in my model, and instead maybe just use a Resource entity that just defines a 'location'. But as a disadvantage, this approach consumes additional I/O due to having to read-then-write to a file in the web layer, then later read from that file when finally saving to the storage, at least.

On the other hand, directly propagating the "stream" would allow full end-to-end streaming, which is extremely more efficient than using files (which also need to be properly deleted, additional temp storage provided, avoid loading fully in memory, etc.). The only disadvantage I see here is that a type like InputStream (in Java) may be considered as 'technical leak' and it's not serializable, in the cases where the final logic to save it is distributed.

What do you think? How do you handle these use cases? Loving to hear feedback from you!

[Sairyss]

Seems like you are deciding between architectural purity vs performance optimizations?

Cannot tell much without seeing full picture and code, not sure why you need to propagate files through all layers, you could just propagate your InputStream directly where you want it to (from web controller directly to infrastructure layer that sends it to the file storage in your case? can't tell without seeing the code), bypassing the layers in the middle.

Usually when I do functionality that includes saving files, I don't create any "middle layers" like domain, I just receive the file in controller and send it directly to the infrastructure that handles saving it somewhere.

On the other hand, if you need that domain layer to save some additional data about the file, you could just send only that data to the domain layer, and the file itself directly to the infrastructure, avoiding sending the file itself through all layers.

Not sure if that helps, if not I'll need more details to understand the problem better.

[emedina]

Yes, that makes sense. I should not get into the domain unless I'm gonna do something with it, indeed.