Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Global workqueue exception adding new work for ACDC workflows #11681

Closed
amaltaro opened this issue Jul 28, 2023 · 1 comment · Fixed by #12204
Closed

Global workqueue exception adding new work for ACDC workflows #11681

amaltaro opened this issue Jul 28, 2023 · 1 comment · Fixed by #12204
Assignees
@amaltaro
Labels
BUG High Priority WorkQueue

Comments

@amaltaro
Copy link
Contributor

Impact of the bug
Global WorkQueue

Describe the bug
While checking the Global WorkQueue logs (specifically @@@), I found a few workflows hitting the following exception:

2023-07-28 10:09:05,634:INFO:WorkQueue:Trying to add more work for: wangz_ACDC0_Run2023C_EGamma1_PromptNanoAODv11p9_v1_230723_163422_6833
2023-07-28 10:09:05,634:ERROR:WorkQueue:Generic exception adding work to WQE inbox: {'Status': 'Running', 'TeamName': 'production', 'WMBSUrl': None, 'RequestName': 'wangz_ACDC0_Run2023C_EGamma1_PromptNanoAODv11p9_v1_230723_163422_6833', 'StartPolicy': {'SplittingAlgo': 'ParentlessMergeBySize', 'policyName': 'ResubmitBlock', 'SliceType': 'NumberOfFiles', 'SliceSize': 1, 'OpenRunningTimeout': 5184000}, 'EndPolicy': {'policyName': 'SingleShot'}, 'OpenForNewData': True, 'Inputs': {}, 'ProcessedInputs': ['/acdc/wangz_ACDC0_Run2023C_EGamma1_PromptNanoAODv11p9_v1_230723_163422_6833/:pdmvserv_Run2023C_EGamma1_PromptNanoAODv11p9_v1_230510_150836_4481:DataProcessing:DataProcessingMergeNANOEDMAODoutput/0/36'], 'RejectedInputs': [], 'PileupData': {}, 'ParentData': {}, 'ParentFlag': False, 'Jobs': 0, 'SiteWhitelist': [], 'SiteBlacklist': [], 'Dbs': None, 'ParentQueueId': None, 'Priority': 0, 'SubscriptionId': None, 'EventsWritten': 0, 'FilesProcessed': 0, 'PercentComplete': 100, 'PercentSuccess': 50, 'TaskName': None, 'ACDC': {}, 'ChildQueueUrl': None, 'ParentQueueUrl': None, 'NumberOfLumis': 0, 'NumberOfEvents': 0, 'NumberOfFiles': 0, 'NumOfFilesAdded': 0, 'Mask': None, 'TimestampFoundNewData': 1690130808, 'NoInputUpdate': False, 'NoPileupUpdate': False, 'CreationTime': 1690130808.2826047, 'WMSpec': <WMCore.WMSpec.WMWorkload.WMWorkloadHelper object at 0x7fe26c2a12e0>, 'Task': None}. Error: local variable 'rejectedWork' referenced before assignment
2023-07-28 10:09:05,635:INFO:WorkQueue:Added 0 new elements for request: wangz_ACDC0_Run2023C_EGamma1_PromptNanoAODv11p9_v1_230723_163422_6833

Note that global workqueue is running version 2.2.2.1

How to reproduce it
Not sure at the moment

Expected behavior
First, ACDCs have only one workqueue element, so I think we should not even try to add work to ACDC workflows.
Nonetheless, what is really expected with this ticket is to correct the assignment of rejectedWork variable to avoid this exception.

Additional context and error message
None
@mapellidario mapellidario mentioned this issue Nov 27, 2024
Closed
@amaltaro
Copy link
Contributor Author

Note this another ticket created by Ahmed on the same issue (and more rich in content):
#12188

@amaltaro amaltaro self-assigned this Dec 11, 2024
@amaltaro amaltaro moved this from ToDo to In Progress in WMCore quarterly developments Dec 11, 2024
@amaltaro amaltaro mentioned this issue Dec 11, 2024
Merged
@github-project-automation github-project-automation bot moved this from In Progress to Done in WMCore quarterly developments Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amaltaro amaltaro

Labels
BUG High Priority WorkQueue
Projects
WMCore quarterly developments
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix WorkQueue broken reference to rejectedWork and badWork amaltaro/WMCore
1 participant
@amaltaro