You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Good day! I have a huge file collection(about 3 million files) and have uploaded around half of it but I stop midway because of some issue(s) in the EC2 server.
Now, I tried to restart the process and I understand that it will again compare and skip out all the unmodified file(s) but it seems like it is taking a lot of time on just listing and comparing the files and I am curious if this is normal? I am currently at 400000 files as listed in the logs and this has been running close to two days already.
I have the following command executed just for context:
"nohup bash -c './mtglacier sync --config config.d/sync-media-backup.cfg --max-number-of-files 3000000 --concurrency 4 --partsize 64 --new >> /var/log/aws-glacier/sync.log' > /dev/null 2>&1 & echo $!"
Additionally, the EC2 server I have is also newly created and is dedicated just for this glacier upload operation. Is their a configuration option that I am missing to somehow throttle the sync process? Appreciate any feedback!
The text was updated successfully, but these errors were encountered:
Good day! I have a huge file collection(about 3 million files) and have uploaded around half of it but I stop midway because of some issue(s) in the EC2 server.
Now, I tried to restart the process and I understand that it will again compare and skip out all the unmodified file(s) but it seems like it is taking a lot of time on just listing and comparing the files and I am curious if this is normal? I am currently at 400000 files as listed in the logs and this has been running close to two days already.
I have the following command executed just for context:
"nohup bash -c './mtglacier sync --config config.d/sync-media-backup.cfg --max-number-of-files 3000000 --concurrency 4 --partsize 64 --new >> /var/log/aws-glacier/sync.log' > /dev/null 2>&1 & echo $!"
Additionally, the EC2 server I have is also newly created and is dedicated just for this glacier upload operation. Is their a configuration option that I am missing to somehow throttle the sync process? Appreciate any feedback!
The text was updated successfully, but these errors were encountered: