Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the possibility to add a random sleep when storing remote datasets #40

Open
d-consoli opened this issue Jul 11, 2023 · 1 comment
Open

Comments

@d-consoli
Copy link

d-consoli commented Jul 11, 2023

Hi, thanks for your work, it is a really useful tool! Unfortunately, when using it to download datasets from Zenodo that contain several (small) files, Zenodo receives too many requests and stops the download. This can be easily solved, for instance, by adding a random sleep in the for loop for downloading the remote datasets, and this is what I did in my fork of your project in this commit. If you think it can be a useful feature and you agree in the way I implemented it I can add an ad-hoc test and create a pull request to your repository, or alternatively search for a better solution. Thanks for your help!

@girgink
Copy link
Contributor

girgink commented Jul 15, 2023

Hi @d-consoli, thanks for suggestion! Surely it is a useful feature. For another project I developed a similar code in the past, which waits for a random time period and does this in a loop, so that if the waiting time is not sufficient it waits for a longer time until the file becomes available or a maximum number of tries is reached. I can add it to the store method. Meanwhile please feel free to create a pull request for your solution if you wish.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants