Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow listing operation during sync operation #144

Open
chrisjob opened this issue May 8, 2019 · 0 comments
Open

Slow listing operation during sync operation #144

chrisjob opened this issue May 8, 2019 · 0 comments

Comments

@chrisjob
Copy link

chrisjob commented May 8, 2019

Good day! I have a huge file collection(about 3 million files) and have uploaded around half of it but I stop midway because of some issue(s) in the EC2 server.
Now, I tried to restart the process and I understand that it will again compare and skip out all the unmodified file(s) but it seems like it is taking a lot of time on just listing and comparing the files and I am curious if this is normal? I am currently at 400000 files as listed in the logs and this has been running close to two days already.

I have the following command executed just for context:
"nohup bash -c './mtglacier sync --config config.d/sync-media-backup.cfg --max-number-of-files 3000000 --concurrency 4 --partsize 64 --new >> /var/log/aws-glacier/sync.log' > /dev/null 2>&1 & echo $!"

Additionally, the EC2 server I have is also newly created and is dedicated just for this glacier upload operation. Is their a configuration option that I am missing to somehow throttle the sync process? Appreciate any feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant