Downloading to external storage on Android

[781flyingdutchman]

The issue

Android developers would like the option to download directly to a external storage (an SD card), presumably for two reasons:

1. Internal storage may be limited, so a large file may not be downloadable, even though it would fit on the SD card
2. The current approach of moveToSharedStorage introduces an extra copy step which is inefficient

Desktop developers would like an option to download to any file path, not only those provided through BaseDirectory and SharedStorage.

Considerations

There are 3 concerns to address:

1. The path to a file is not stable in iOS, which is why we use baseDirectory, directory and filename to determine the location of a file. Creating an option to use an absolute path therefore is confusing and will likely lead to hard-to-fix bugs, especially on iOS. The path to external memory on Android is also not stable (e.g. an SD card can be removed) but from the feedback it seems that this is considered less of an issue, so one we will ignore here. It does mean that downloads to external memory will be less confident and may fail when started.
2. Android API 30+ uses the MediaStore to get URIs pointing to 'shared storage', which is why we've introduced the SharedStorage enums. However, these can only be used in method moveToSharedStorage, as a second step after downloading a file - they cannot be used as download destinations for the initial download. In addition, the introduction of the MediaStore means you can no longer use an absolute path in Android either (which is what the Desktop developers would prefer)
3. We cannot create the download destination file directly, because the downloader 'contract' says that only successfully completed downloads will result in a file existing at the destination. We therefore need a temporary location for the file to download, then move (or in practice: rename) the file to its final destination file name & location.

For posterity: the reason we use baseDirectory, directory and filename to determine the location of a file (instead of a much simpler full path) is that paths on mobile are not stable. iOS, every time you start the app it will get a new sandbox, with a different base path. Because the downloader needs to work for all platforms similarly, this requires us to split the file location in three.

Possible solutions

There doesn't appear to be a way to pass a full absolute path to Task in a way that works for all platforms, plus it has significant drawbacks. The Desktop developer that needs this can always choose to rename or move a downloaded file to wherever they want using plain Dart code, so I am going to suggest we ignore this request and pursue the route using the SharedStorage enum as a way to indicate where files should go, and solve for the Android external storage use case only.

To solve for the external storage directory in Android, we could:

1. Add a value BaseDirectory.root to the BaseDirectory enum, such that the path provided by directory and filename effectively becomes a path from root to anywhere. This would satisfy the Desktop developers, but would a) create issues for iOS developers and b) is not implementable in Android because of the MediaStore.
2. Add multiple values to the BaseDirectory enum that would equate to external storage on Android. However, this would be very confusing for non-Android developers as those have no true equivalent on those platforms, and appear as duplicates of the SharedStorage enum.
3. Add a boolean field external to Task just for Android that if true applies the directory parameter to the external directory. This helps somewhat, but the choice of directories within external storage is not unlimited, and we still have the issue with the MediaStore, which expects files to fall in certain categories
4. Add a SharedStorage field external to Task just for Android that, if not null, forces the download to the appropriate external storage directory. We then need to update the Task.filePath method to return the proper file path as returned by the MediaStore.
5. Add a configuration option for Android only, to effectively change one of the BaseDirectory enums into one of the SharedStorage destinations. This would allow the same functionality as 4, but set 'permanently' for all future downloads, and without the need for an extra field in Task.

I'm leaning towards solution 4 as the preferred approach, with options for the field: external, androidExternal, sharedStorage, androidSharedStorage and perhaps others.

I also like 5, as it's simple and explicitly requires Android developers to make the choice to change how the downloader works.

To solve for the need to not download to a temporary file in internal storage, we could create a random temporary file in external storage (using the MediaStore) and update the MediaStore with the desired file name once the download completes successfully. We would only do this if the download destination is an external destination on Android - it would not change normal functionality.
Alternatively, we could drop the contract for this type of direct-to-external download, meaning we don't create a temporary file and instead download straight to the desired external destination.
Alternatively we could add a configuration option to activate this direct-to-external download.

[nahidmk]

May I have some examples of solution number 4 and 5 ?

[781flyingdutchman]

For 4, a task would be created like this:

final task = DownloadTask(
    url: `https://google.com`,
    external: SharedStorage.video);

This will download the file directly to the URI given by the MediaStore: MediaStore.Video.Media.getContentUri(MediaStore.VOLUME_EXTERNAL_PRIMARY). It is my understanding that that path is the primary external storage (though there appear to be possible multiple external storages, so still not 100% sure this gets the files on - for example - a mounted SD card).

For 5, the task would not change, and instead upon startup of the app you would configure, eg:

FileDownloader().configure(androidConfig: (Config.mapToExternal, (BaseDirectory.applicationSupport, SharedStorage.video));

This would configure the downloader to map the BaseDirectory.applicationSupport to SharedStorage.video for all future downloads, so any task that has its baseDirectory set to BaseDirectory.applicationSupport on Android would now store in the shared storage for video (same URI as for 4 above). The benefit of this is that it requires no change to Task and no further change in code for Android, provided you can map your downloads to all go to the same SharedStorage destination. The downside is that it can be a bit confusing, as you're not really downloading to BaseDirectory.applicationSupport so it may lead to confusion.

[781flyingdutchman]

Option 3 (a boolean external added to Task) could also be attractive, in that it simply stores the file to the directories suggested by baseDirectory but instead of storing them in the directory returned by context.filesDir and context.cacheDir we use context.externalFilesDir and context.externalCacheDir. Everyting else stays the same, and you'd still have to use moveToSharedStorage if you want to move the file out of applicaton-specific storage. But in practice, if the file is already in external storage then a) there is much more space available, and b) a 'move' is really a rename, so is very fast (it doesn't actually copy the file).

There is a variant (let's call it option 3a) which uses a configuration to set Android to use external storage for all downloads, or for downloads greater than a certain size. For example (you'd only use one):

FileDownloader().configure(androidConfig: (Config.useExternal, Config.always);
FileDownloader().configure(androidConfig: (Config.useExternal, Config.never);
FileDownloader().configure(androidConfig: (Config.useExternalIfFileLargerThan, 100);  // if >100MB file

Option 3a has the advantage of not requiring any change to Task and being a true Android configuration.

The more I think about it, the more I think option 3a is the simplest and most consistent of all options.

[781flyingdutchman]

@nahidmk this is now available on the dev branch of the repo. Any change you can try this out? See here for how to configure this, and the limitations. Please let me know if this works and addresses your needs.

[781flyingdutchman]

Implemented in V7.11.0