-
Notifications
You must be signed in to change notification settings - Fork 484
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Globus integration: further improvements/refactoring #10623
Comments
also, I'll just mention the Globus Dataverse App code is now GDCC's Github repository: https://github.com/gdcc/dataverse-globus and any app related issues (including ongoing internationalization efforts) will be recorded there. |
Mentioning this here as we have been doing a lot of testing of uploads of datasets with many files (ie: 3000+) and have run into an issue with the CURL command to upload to Dataverse via API. We tried with various subsets of the files (ie: 15, 300, 500, 700, 900) and found that the upload to Dataverse using Globus was successful up to somewhere between 400-500 files. In every instance the files were successfully uploaded to the S3 bucket via Globus (even the full 3000 file load) but for those over 500 files the call to register to Dataverse failed and the dataset lock never cleared. We had to manually delete the lock and delete all the files from the bucket. Examination of the Globus logs showed that the Globus uploads were successful, but the call to Dataverse with that many files failed due to the CURL command call being too long for the shell environment. The error that I see in the Globus log is:
Although the files go on to successfully upload to the bucket via Globus. They never actually register in Dataverse. |
Thank you for the report @nightowlaz! |
… some cleanup and refinements. #10623
Resolved conflicts: src/main/java/edu/harvard/iq/dataverse/api/Datasets.java (#10623)
A placeholder issue for keeping track of next steps in improving Globus support after the conclusion of the initial integration effort.
At the moment I am aware of a couple of areas where improvements can be made. This is all in the old code that was originally created by the Scholars Portal. While these are not bugs or something that's broken outright, there is a possibility that this refactoring can make the workflow more reliable, especially on a busy prod. server.
/addFiles
API call to itself, via curl. Once again, this is working in the current implementation and we are not aware of any real problems because of this. But it just would be much cleaner for the service to call the appropriate commands/methods directly.Related
The text was updated successfully, but these errors were encountered: