Skip to content
This repository has been archived by the owner on Feb 24, 2022. It is now read-only.

# Components description

dciangot edited this page Jun 12, 2018 · 2 revisions

State machine

The transfer state machine is all in the transfer DB backend (currently Oracle) and retrieved via REST call to the CRAB interface on cmsweb.

AsyncTransfer

The Asynctransfer component is responsible for the following actions:

  • retrieve output files and logs to be transferred
  • group files by user and then by link (source, destination)
    • prepare one transfer job per link
  • start a thread per user
    • update status to "acquired"
    • retrieve user proxy from myproxy (where CRAB put at submission time)
    • delegate proxy to FTS3
    • for each link submit a FTS job
    • update the status on the DB as "submitted"
    • save jobID and other metadata to a sanbox in the local filesystem

Monitor

The Monitor component is responsible for the following actions:

  • periodically poll the job status from FTS for all the jobID on the local filesystem
  • when the status is terminal, dump the transfer results on another local sanbox for the Reporter

Reporter

The Reporter component is responsible for the following actions:

  • look into the sandbox for ended transfers
  • update the status on the DB based on the transfer exit code (one thread per user):
    • DONE for successful transfers
    • RETRY if the transfer failed and the retry count is less that 3. (also allowed only for certaing categories of error)
    • FAILED otherwise

DBSpublisher

The DBSpublisher component is responsible for the following actions:

  • get files that have been transferred succesfully and that the CRAB postjob marks as "to be published"
  • group them by user
  • launch one thread per user:
    • for each task:
      • update publication status to "publication acquired"
      • retrieve file metadata from CRAB cache
      • wait if the task is not in a final state or a certain amount of time is passed since the last published file of this task
      • publish files in DBS
      • update the publication status to "done" or "failed" accordingly

RetryManager

The RetryManager component is responsible for the following actions:

  • set transfer status to new after some time for files in transfer status "RETRY"
  • kill transfer jobs using FTS rest for all files marked by CRAB as "KILL"