[FEATURE REQUEST] - Add storage limit check to file migration plugin (ideally to core too) "[FEATURE REQUEST]" .

[tomekduda]

Is your feature request related to a problem? Please describe.

I am seeding a developer sandbox (200 MB data storage, 200 MB file storage) with production data. I'm using core:ExportFiles, the (slightly anonymised) relevant bit of export.json is below, I'm already limiting files to 1 MB size

{
    "operation": "Upsert",
    "externalId": "Name",
    "query": "SELECT all FROM FileApproval__c",
    "master": false,
    "afterAddons": [
        {
        "module": "core:ExportFiles",
        "args": {
            "operation": "Upsert",
            "maxFileSize": 1048576,
            "deleteOldData": true
        }
        }
    ],
    "deleteOldData": true
}

Of course there's no total cap so after a while the storage starts to skyrocket:

And logs indicate the plugin dutifully tries to load:

WARNING: [13:52:45.239] [WARN] [Batch# REST:Insert] {ContentVersion} Completed with issues. 17 records processed, 17 records failed.
[13:52:45.242] {ContentVersion} The Insert API operation has been completed.
[13:52:45.360] [core:ExportFiles] {FileApproval__c} Processing chunk #81 of 15 items.
[13:52:45.903] In progress... Completed 0/15 records.
[13:52:46.393] In progress... Completed 10/15 records.
[13:52:47.124] In progress... Completed 15/15 records.
[13:52:47.128] {ContentVersion} The Insert API operation has been started using REST API.
WARNING: [13:52:55.103] [WARN] [Batch# REST:Insert] {ContentVersion} Completed with issues. 15 records processed, 15 records failed.
[13:52:55.108] {ContentVersion} The Insert API operation has been completed.

Describe the solution you'd like

I don't know what's easier / makes most sense?

• Perhaps a new config option for the core:ExportFiles to stop after inserting X megabytes? Default zero meaning no limit?
• "stop after using 95% of the org storage"?
• or if it detects a 100% fail rate - can it run a REST call? sf api request rest 'services/data/v62.0/limits' for my situation returns

"DataStorageMB": {
  "Max": 200,
  "Remaining": 182
},
"FileStorageMB": {
  "Max": 200,
  "Remaining": -69
}

It wouldn't be perfect as storage statistics have couple minutes delay (so I imagine even detecting "95% reached" would not work in real life) - but it'd beat waiting through a step I know won't get any better.

[hknokh]

Hello, @tomekduda

Thank you for your feature request.
I will review it as soon as possible and provide updates as they become available.

Cheers

[github-actions]

This case has been marked as 'to-be-closed', since it has no activity for the 3 days.
It will be automatically closed in another 3 days of inactivity.

[hknokh2]

Hello, thanks for the idea.
I will put it on the track for the future development.
For now I don't plan to add new features to the project, but may be later.
Cheers.

[github-actions]

This case has been marked as 'to-be-closed', since it has no activity for the 3 days.
It will be automatically closed in another 3 days of inactivity.