-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use pooch
for downloading the model:
#36
Comments
FYI, i've found that SHA256 is quite slow at computing hashes for large files. (i.e. 2GB models...) you might want to play around with maybe cryptographically non-secure hashes. I've used XXH128 with great success. |
Hi @hmaarrfk, I think that would make sense. I don't think we require the hashing to be secure since the goal here is not to avoid malicious attacks, but just to make sure that files do not change. In case you want to look into this: contributions are very welcome. (If not: no worries, but this would be on my list of lower priorities, so I can't really promise when we'd have time to look into this). |
That should be straightforward to do, pooch supports the python-xxhash bindings. The docs even have this nice example about how to use it, you just append import datetime
import pooch
# Get the current data to store the files in separate folders
CURRENT_DATE = datetime.datetime.now().date()
GOODBOY = pooch.create(
[...],
registry={
"store.zip": "xxh128:6a71973c93eac6c8839ce751ce10ae48",
},
) |
Implemented in #276 |
Originally posted by @constantinpape in #35 (comment)
The text was updated successfully, but these errors were encountered: