README

* Copyright (c) <year>, <copyright holder>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*     * Redistributions of source code must retain the above copyright
*       notice, this list of conditions and the following disclaimer.
*     * Redistributions in binary form must reproduce the above copyright
*       notice, this list of conditions and the following disclaimer in the
*       documentation and/or other materials provided with the distribution.
*     * Neither the name of the <organization> nor the
*       names of its contributors may be used to endorse or promote products
*       derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY CINECA ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL <copyright holder> BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

# Author: Giacomo Mariani

Requirements
------------

* Python 2.7
* python-m2crypto 
* Globus Online API 
* m2.py [1]
* grid-proxy-utils (in particular grid-proxy-init)
* myproxy commands (in particular myproxy-init)

Optional:
* iRODS icommands

[1] You should replace the file m2.py installed with the GO API.

Installation
------------

* download all the needed files: 
  $ git clone https://github.com/EUDAT-B2STAGE/B2STAGE-DataStaging-script.git
* install globus online api, for example:
  $ pip install globusonline-transfer-api-client

Optional:
* configure your irods environment in order to point to the iRODS server enabled 
  with the needed rules


The script consists of the following files:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* datastager.cfg.template template for the required (cp to) datastager.cfg file

* datastager.py the main script
* datamover.py the script which invokes GO 

* example some concrete example
* pid.file.example an example of PID file, i.e. a list of PIDs (to be used with -PF option)
* task.file.example an example of path file, i.e. a list of file names (to be used with -pF option)
* url.file.example an example of URL file, i.e. a list of URLs (to be used with -UF option)
* README some further informations

Optional:
* PIDselecter.r the rule which gives you the PID of a file given its URL
* URLselecter.r the rule which gives you the URL of a file given its PID

The script works as follow:
~~~~~~~~~~~~~~~~~~~~~~~~~~~

* calculate the list of files to be transferred (for example, extracting the iRODS path from the PID)
* activate the endpoints involved in the transfer in the following way 
* auto-activation (good for globusconnect)
** if not myproxy activation (if you have a myproxy server associated with your enpoint)
** if not local activation (which requires python-m2crypto package) and uses your local proxy
* delegate GO to do the transfer

Usage
-----

See: 
$ ./datastager.py -h 

The script can run interactively if invoked as:
$ ./datastager.py -I


Troubleshooting
---------------

Error
~~~~~
IOError: [Errno 2] No such file or directory: '/path/to/datastagerconfig.py'
Solution
~~~~~~~~
Copy datastagerconfig.py.example to datastagerconfig.py and edit it accordingly to your
system

Error
~~~~~
globusonline.transfer.api_client.ClientError: ClientError.NotFound (404
Not Found): Subtask details are not available for this task, use the
successful_transfers API instead.
Solution
~~~~~~~~
Upgrade your installation to, at least, release-2.0

Error
~~~~~
globusonline.transfer.api_client.ClientError: ClientError.AuthenticationFailed
(400 Authentication Failed): You must authenticate to this service using a signed
cookie from www.globusonline.org or an x509 certificate associated with your CLI
account and an x509 username cookie or header. If you just created your account,
your account may still be propagating, so please try again in a minute.
Solution
~~~~~~~~
Check that the public part of the proxy (usercert.pem) is associated in GO and
that you are using the right username (the one used to login to GO). 

Error
~~~~~
When using GC as endpoint the file does not exist.
Solution
~~~~~~~~
GC put the files in the home of the user so the path must be relative to that home
directory.