Request Status
This page describes the different status for a request in ReqMgr.
A new request, this status is usually skipped and the request goes directly to assignment-approved when created with script tools. (McM)
Requests in this status are awaiting review and assignment from the CompOps L2s. They will be moved to rejected if there is a problem with the request, otherwise it is assigned. (McM, Physics groups)
Assigned requests have been reviewed and modified by the CompOps L2s, these requests have been provided with an appropiate site whitelist, acquisition era, processed dataset and other attributes. Requests in this state are examined by the WorkQueue and moved to acquired after creating the work elements, or in case of failure the request is moved to failed. (Unified, Ops)
Acquired requests have been split by the global WorkQueue into work elements, but no work element has been injected into the Local Workqueue of any WMAgent and therefore not considered as running yet. (ReqMgr2)
Refactored on 10/Nov/2021: Requests will be set to running-open state when at least one WQE has been pulled by an agent (thus, in state Acquired or beyond). It also requires to have other WQEs waiting in Global Workqueue (either in Available/Negotiating/Acquired state, thus not yet in WMBS).
The old behavior of running-open requests required to have at least one work element injected into Local Workqueue (having jobs created or not). It is running-open for historical reason when DBS supported open blocks and data still can be added but this definition is not valid anymore since DBS3 only allows closed blocked (ReqMgr2, via GlobalWQ elements' status checking)
Requests will be set to running-closed state when all of the WQEs are in at least Running state, thus everything has been injected into WMBS and there are no workqueue elements available in GQ.
Set by user which will kill all remaining work. But workflow will still move to completed status. (Ops, Unified)
A request is marked as completed after all work elements are done, which means that the WMAgent(s) have processed all the jobs generated by each one of them. This includes not only the top level task, but also the auxiliary ones like log collection and cleanup of unmerged data. A completed request will be looked at by CompOps people to verify the success or failure of it, when the output of the request is considered satisfactory the request is moved to closed-out status, otherwise to rejected. (ReqMgr2, via GlobalWQ elements' status checking)
Note that a request in completed is not guaranteed to have all its output data registered in DBS and/or PhEDEx, although this is usually taken for granted there are failure cases when this may not happen automatically.
Closed out status indicates that the output has been reviewed and is ready to be announced back to the requestors. (Unified)
An announced request has been announced to the requestors using the usual channels and can be archived. (Unified, Ops)
After a request is announced the WMAgent takes care of cleaning up most of the monitoring information from the system about it and then it is marked as normal archived. (ReqMgr2, via WMStats job information and archive delay config parameters)
A request is moved to rejected when it is considered invalid at assignment or when the produced output is not satisfactory. (Unified, Ops)
After a request is rejected the WMAgent takes care of cleaning up most of the monitoring information from the system about it and then it is marked as rejected archived. (ReqMgr2, via WMStats job information and archive delay config parameters)
A failed request has had a failure in one of the work elements, or it didn't produce any. These can be re-evaluated and reassigned to run again, or move to rejected state if unrecoverable. (GlobalQueue)
If there is an unrecoverable problem with a request after it has been acquired, then it is possible to move it to aborted state. This will trigger an internal action to kill all current jobs and run only auxiliary tasks like unmerged data cleanup and log collection, after all these actions are completed the request will be moved to aborted completed. (Ops)
A request is marked as aborted completed after all left-over jobs have been processed in an aborted request, a request in this state has been cleaned up from the WMAgents and global WorkQueue and is ready to be archived.(ReqMgr2, via GlobalWQ elements' status checking)
A request is moved to aborted-archived after it is aborted-completed and all monitoring information related to it has been cleaned up. (ReqMgr2, via WMStats job information and archive delay config parameters)
Common issues among different states, and their solution:
- Issue: Too many workflows stuck in assigned state
Problem: team is not set properly. A block is going to be pulled by an agent only if the team is set properly. The teams are: mc, mc_highprio, repro_lowprio, repro_highprio, step0, hlt, relval. Solution: elog about the problem, the workflow has to be resubmitted.
Problem: Site is not assigned properly. Some workflows need to be submitted to specific places (i.e. ACDCs) Solution: There is no one solution to this, you have to check each case. If the site where the block is available is not set in the whitelist, send an elog about this.
Problem: White list only includes the _Disk of a site. Solution: For sites where disk and tape are separated, elog about this and recommend to send the request to both T1_XX_XXXX and T1_XX_XXXX_Disk. Read this
- Issue: Too many workflows acquired and not moving in an agent
Problem: