Potential challenge verifying the hash of data

[brando90]

Sorry I wasn't sure if to make this an issue or a question but I assumed that a question might make it more visible for future users.

It might be difficult for checking the hash of large files. e.g. I get this error when checking the hash for the data:

Traceback (most recent call last):
  File "/Users/brando/CoqGym/unzip_data.py", line 42, in <module>
    check_md5('data.tar.gz', '922937155a199605eb8067ccfbbdb81a')
  File "/Users/brando/CoqGym/unzip_data.py", line 19, in check_md5
    hashcode = md5(open(filename, 'rb').read()).hexdigest()
OSError: [Errno 22] Invalid argument

when running python unzip_data.py.

I saw there were some solutions (which might not be worth implementing) but just in case this is useful in the future. I am currently just not verifying the hash for the data.

https://stackoverflow.com/questions/48122798/oserror-errno-22-invalid-argument-when-reading-a-huge-file

[brando90]

Currently, I am just not verifying the hash.

Potential better solutions check: https://stackoverflow.com/questions/48122798/oserror-errno-22-invalid-argument-when-reading-a-huge-file

feel free to improve my answer if you have a better solution :)

[yangky11]

Thanks Brando! I think the solution on StackOverflow is great. We could implement it in the check_md5 function.

I also think that MD5 check failure should be treated as a warning rather than an error. It just means the data is different from the official data, which is not necessarily an error.

[yangky11]

I updated unzip_data.py to implement the method

[brando90]

alsi I wrote a script for downloading the data from gdrive automatically (usueful for HPCs, I can share when I test it properly - or share it anyway and we can see what happens) Thanks for your openness Kaiyu! Sincerely, Brando

…

On Jan 24, 2021, at 8:24 AM, Kaiyu Yang ***@***.***> wrote: I updated unzip_data.py to implement the method — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE6LWYTLUI6CCHOENSNW3S3QUQFANCNFSM4WNLQCYA>.