Feature Request: Uploading in Chunks, adding Offset/Length arguments

[hpoul]

When uploading large files (e.g. videos) it could be beneficial to upload them in chunks with separate requests. Currently this could probably be done by locally saving single files locally, essentially duplicating the storage requirement..

I think it would be beneficial to add something like offset and length parameters to the UploadTask which would only upload a certain part of a file in that request, instead of the whole file.

For example AWS supports multi part uploads, and suggests using them for files > 100MB.

[781flyingdutchman]

Hi, thanks for sharing. I did a little bit of research, and as I understand this is an AWS specific technique, not a formal HTTP standard, though Google Cloud seems to also have adopted it. Do you know how widespread the adoption of this technique is? I am hesitant to create something that isn't widely supported, so want to have some evidence this has broad acceptance and adoptions with commonly used upload servers.

[hpoul]

Thanks for investigating.. it doesn't really have much to do with HTTP actually.. it's not like a download where the range header is understood by the HTTP server ..
It's completely transparent to the HTTP layer, and only the application layer would have to know about it.. i think it's a pretty common technique for anything requiring large uploads..
It shouldn't add too much complexity.. it doesn't change anything in the actual http request.. it would simply only include one part of the provided file, instead of the whole file..

[781flyingdutchman]

My initial understanding was that you were looking for an implementation of the S3 multipart upload protocol. After doing some research, I indeed see that this protocol is commonly implemented by cloud storage providers, but I also noted that each implementation is slightly different in the detail (e.g. uses service-specific headers for authentication, pre-signed urls that require use of the service specific SDKs, etc). Based on that I don't think this is a viable route for the core plugin, though I could see value in someone building an extension that can convert a single large upload into the protocol described using a service-specific SDK.

Upon reading your initial comment again, I wonder if what you're asking is only to allow a 'byte range' like parameter for the upload task that would upload only a portion of a larger file, and that you would implement the S3 multi-upload protocol yourself at the application layer, using the enhanced Upload functionality of the downloader. You would be responsible for maintaining state across the multiple uploads, and for checking and sending the completion signals. Is that correct? The completion signal requires you to collect the ETag of each upload - that is not something the Upload task currently passes back, so how would you deal with that?

I wonder if there is a middle ground in the form of a ParallelUpload class that is abstract, and that follows the S3 protocol in that it has CreateMultipartUpload, UploadPart and CompleteMultipartUpload methods that must be implemented by the developer (you!) specific to the service you are using. You would be responsible for setting the right headers and url parameters (e.g. creating a pre-signed URL) and the ParallelUpload object schedules the actual uploads by calling UploadPart repeatedly, collects the response information, tracks state & completion and calls the CompleteMultipartUpload method with the list of ETags when complete. This would allow flexibility for multiple providers, some standardization of the approach, and relatively minor changes to the native side of the upload logic.

It is also possible that we could create concrete versions of the abstract ParallelUpload class for S3 and GCP for instance, but this requires a dependency on those respective SDKs and I am not sure that those are 'tree-shaken' out if a developer doesn't use this logic (which will be the vast majority of developers using the plugin). We could potentially put those concrete versions in a separate package (just like we have done for SQL support in package background_downloader_sql).

Let me know what you think.

[hpoul]

I wonder if what you're asking is only to allow a 'byte range' like parameter

Yes exactly, that was actually what I was asking for. But I did not realize that it's not possible to access the headers of a completed upload.

I don't think it's necessary for the core package to directly add full support for the AWS HTTP Protocol.. But making it possible would be nice.

ie.

1. adding support for defining the range of a file which is sent as body of a request and
2. returning the HTTP headers of all successful uploads.

I don't really see a reason why it would be required to add the CreateMultipartUpload and CompleteMultipartUpload to the core package. I guess this could be also implemented fairly easily in dart code by the user of the plugin, or by a wrapper package.. But maybe I'm still missing something.

[781flyingdutchman]

Apologies - I am learning about S3 and multi-part uploads as we go here. I found Amplify documentation for uploads, and it seems like this is taking care of all of the complexity related to authentication etc, and actually uses multi-part uploads when the file size is > 5Mb. Similar functionality exists for GCP using Firebase (though not sure if it switches to multi-part uploads).

My question then is: why do you need to use the background_downloader? Is using the Amplify approach not working, or does it have limitations?

Adding a range selector and returning the response headers (in addition to the response body) may not be a bad idea anyway (I'll look into that after I've finished another improvement I am working on) but I am trying to understand why you would want to go to the low-level implementation of the multi-part upload yourself.

I don't really see a reason why it would be required to add the CreateMultipartUpload and CompleteMultipartUpload to the core package.

I see a lot of developer struggle with S3, and multi-part upload is more complex, so I thought perhaps this is an opportunity to streamline that by offering a standardized way to create a MultiPartUpload task that behaves similar to a normal Upload task (e.g. gets status updates, progress updates, etc), where the developer only has to 'fill in the blanks' depending on the cloud provider they are using. It would have some benefits over doing it yourself (e.g. the completion logic would execute on the native side, also when the app is suspended, and state management and progress updates would be automated). I could see a second example app that illustrates how to use the plugin with AWS and GCP using this construct, and perhaps adding subclasses of the regular DownloadTask and UploadTask that leverage Amplify and Firebase plugins to simplify working with the cloud provider. But, I may be overthinking this.

[hpoul]

Sorry for the confusion. I am not actually trying to use S3, it was just an example that it's a common practice. I'm actually implementing a more obscure video upload API which essentially works the same way.. Although in this case my own backend would generate those signed part URLs and send those to my flutter app. And my flutter app would upload the video file to those signed part URLs, collect the etags and once all completed send them back to my backend. My backend would than forward it to that video hosting service..

Right now i've implemented it with the http package.. but I'd prefer having background uploads with more reliable progress indication and not requiring the user to keep the app open.

[781flyingdutchman]

Got it. I've moved this to the 'Discussions' tab under 'Ideas' and plan to implement this in a future version. I'm currently focused on a few other things, so this may take some time. Thanks for your patience.

[781flyingdutchman]

@hpoul I have good and bad news. Good news is that V8.1.0 now returns responseHeaders as part of the final TaskStatusUpdate (just like it does for the responseBody). Note the header names are lower-cased, in line with the Dart convention. Bad news is that partial file upload is not possible in a background task on iOS. To do this you would need to create an uploadTask(withStreamedRequest:) instead of an uploadTask(with:fromFile:) that we use today, and all documentation (and my experience, as I implemented this to try) says that those streamedRequests only work in the foreground. Because I want the performance to be consistent across all platforms, I cannot implement partial file upload, so you will have to create the chunks in separate files and submit those as regular uploads. I realize this may double your storage requirement, but there is no other solution that I am aware of.

I'm going to close this discussion, but happy to elaborate further if that's helpful.

[hpoul]

I actually thought the whole point of the UploadTask was to upload data in the background..
Would it work on Android? Maybe it would be possible than to make an iOS implementation which does it manually (ie. creating separate files for each chunk before creating the upload tasks .. which is what i would do otherwise in flutter.. but I guess it would be beneficial if it would be done in native code, and could be skipped on android 🤔)

[781flyingdutchman]

It is, and it does exactly that for a file upload, but for some reason it cannot handle a stream upload in the background. It's likely possible in Android and Desktop, but "simulating" that in iOS adds a level of complexity I'm not quite ready for, as I still consider this rather niche. At least with the responseHeaders field you should be able to build your chunked upload mechanism, even if still at the cost of extra disk space. You could consider building a mechanism where only a portion of the chunks are created and uploaded at any time, so you don't fully duplicate the original file (as you can delete the chunk files when their upload completes).

[781flyingdutchman]

One more clarification (as you got me thinking about this alternative iOS implementation!): are these file uploads binary uploads (i.e. you set the post field of the UploadTask to 'binary') or do you use multi-part uploads (not the kind S3 uses, but the form-data kind)? I assume (and hope) it is binary, as that is much easier, but let me know.

[hpoul]

I think there is some confusing terminology.. S3 seems to refer to uploading one file in multiple requests as "multipart upload" .. while this has nothing to do with the multipart/form-data content-type...
imho those two things are complete opposites.. the "chunked" upload is used to upload one huge file in multiple requests.. while the http multipart/form-data is used to upload multiple small files (and other data) in one request.. I don't think combining those two makes any sense at all... ie. the body of my requests only contain the original binary of one file beginning at different offsets.. (ie. chunked). (although I'm using the content-type of the original file.. which would be video/mp4 for example, even though it's only part of a video file..

[781flyingdutchman]

Yes, that's what I thought. I would indeed refer to that as a binary file upload (where every chunk is a binary upload, and together they make up the big file) and we can stay away from the complexity of multipart/form-data which is indeed more for simple uploads that require additional metadata.

[zhanchen18]

Has there been any progress on this discussion? I am planning to implement what you have been discussing, but I am still in the stage of reading the code. Since I am not familiar with Kotlin, I feel a bit lost at the moment.

[781flyingdutchman]

See #370 for how I am currently thinking of implementing this. I haven't gotten to it, so would appreciate your help, though I want to be clear that in order for a PR to be accepted it needs to follow the suggested in #370 and must be implemented for all platforms (iOS, Android, Desktop) and must have integration tests written for it (in the example directory). Let me know if you want to take this on, otherwise please wait until I get to this.

[zhanchen18]

I wish I could contribute more, but with my current level of Kotlin proficiency, I'm not quite up to the task yet. Could I get an idea of the expected timeline?

[781flyingdutchman]

No worries - should be fairly quick, I got it working today.

[781flyingdutchman]

Implemented in V8.6.0 using "Range" header