Globus integration: further improvements/refactoring

[landreev]

A placeholder issue for keeping track of next steps in improving Globus support after the conclusion of the initial integration effort.

At the moment I am aware of a couple of areas where improvements can be made. This is all in the old code that was originally created by the Scholars Portal. While these are not bugs or something that's broken outright, there is a possibility that this refactoring can make the workflow more reliable, especially on a busy prod. server.

1. After the Globus service receives the confirmation of the completion of the Globus upload, in order to add the file(s) to the dataset it makes an actual /addFiles API call to itself, via curl. Once again, this is working in the current implementation and we are not aware of any real problems because of this. But it just would be much cleaner for the service to call the appropriate commands/methods directly.
2. This one is potentially more important, but may also take more work to implement. As implemented, the asynchronous process of waiting for the Globus upload to complete relies on a service method continuously checking on its status remotely, in a loop (with a sleep interval) for the duration of the process. If nothing else, this means that an upload in progress is not going to survive an application restart before its completion (i.e., the file will end up successfully deposited into the Globus collection, but will not be added to the dataset on the Dataverse side). We need to keep in mind that some of the real life big data uploads will be taking a very long time, days etc., to complete (which is the whole point of Globus etc. etc.). There's also some anecdotal evidence of that looping process having died on the application side while one of the OMAMA data files was being uploaded in prod., even without any restarts involved (possibly on account of something unexpected during one of the intermediate Globus API calls - ?). So, it would make sense to refactor this part as to handle it truly asynchronously. Perhaps by storing the state info about the ongoing uploads in a database table, then have a timer job check on all such known uploads every N seconds.

Related

• Service: Harvard Dataverse Repository Large Data Services dataverse-pm#268

[amberleahey]

also, I'll just mention the Globus Dataverse App code is now GDCC's Github repository: https://github.com/gdcc/dataverse-globus and any app related issues (including ongoing internationalization efforts) will be recorded there.

[nightowlaz]

Mentioning this here as we have been doing a lot of testing of uploads of datasets with many files (ie: 3000+) and have run into an issue with the CURL command to upload to Dataverse via API. We tried with various subsets of the files (ie: 15, 300, 500, 700, 900) and found that the upload to Dataverse using Globus was successful up to somewhere between 400-500 files. In every instance the files were successfully uploaded to the S3 bucket via Globus (even the full 3000 file load) but for those over 500 files the call to register to Dataverse failed and the dataset lock never cleared. We had to manually delete the lock and delete all the files from the bucket.

Examination of the Globus logs showed that the Globus uploads were successful, but the call to Dataverse with that many files failed due to the CURL command call being too long for the shell environment. The error that I see in the Globus log is:

  <message>******* Unexpected Exception while executing api/datasets/:persistentId/add call </message>
  <message>java.io.IOException: Cannot run program "bash": error=7, Argument list too long</message>

Although the files go on to successfully upload to the bucket via Globus. They never actually register in Dataverse.

[landreev]

Thank you for the report @nightowlaz!
Yes, this is definitely one extra reason to get rid of that curl command in the implementation.
I'm going to try and get this issue prioritized asap. I already spoke about it with Stefano here.