Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WMAgent - checkProxy.py script silently fails #12173

Closed
todor-ivanov opened this issue Nov 21, 2024 · 0 comments · Fixed by #12174
Closed

WMAgent - checkProxy.py script silently fails #12173

todor-ivanov opened this issue Nov 21, 2024 · 0 comments · Fixed by #12174
Assignees
@todor-ivanov
Labels
BUG WMAgent

Comments

@todor-ivanov
Copy link
Contributor

todor-ivanov commented Nov 21, 2024 •
edited
Loading

Impact of the bug
WMAgent

Describe the bug
While working on a different issue: #12166, related to the virtual environment cornjobs setup, I figured out the checkProxy.py script silently fails for any setup of ours, being it venv or docker. Once I executed the command manually in one WMAgent docker container I got the error:

(WMAgent-2.3.4.4) [cmst1@vocms0290:current]$ python /usr/local/deploy/checkProxy.py --proxy /data/certs/myproxy.pem
Traceback (most recent call last):
  File "/usr/local/deploy/checkProxy.py", line 164, in <module>
    main(sys.argv[1:])
  File "/usr/local/deploy/checkProxy.py", line 82, in main
    proxyInfo = [line for line in out.split('\n') if line]
TypeError: a bytes-like object is required, not 'str'

Here is the set of cronjobs installed at the container:

(WMAgent-2.3.4.4) [cmst1@vocms0290:current]$ crontab -l 
55 */12 * * * date -Im >> /data/srv/wmagent/2.3.4/logs/renew-proxy.log && /usr/local/bin/manage renew-proxy 2>&1 >> /data/srv/wmagent/2.3.4/logs/renew-proxy.log
58 */12 * * * python /usr/local/deploy/checkProxy.py --proxy /data/certs/myproxy.pem --time 120 --send-mail True --mail [email protected]
*/15 * * * *  source /usr/local/deploy/restartComponent.sh 2>&1 >> /data/srv/wmagent/2.3.4/logs/component-restart.log
* * * * * echo 'http://cmst1:[email protected]:5984/wmagent_jobdump\%2Fjobs/_design/JobDump/_view/statusByWorkflowName?limit=1' | sed -e 's|\||g' | xargs curl -s -m 1 > /dev/null
* * * * * echo 'http://cmst1:[email protected]:5984/wmagent_jobdump\%2Ffwjrs/_design/FWJRDump/_view/outputByJobID?limit=1' | sed -e 's|\||g' | xargs curl -s -m 1 > /dev/null

So the error occurs, because since a while now, the system execution of a sub process from this line here, returns a bytes object rather than a str:

WMCore/deploy/checkProxy.py

Line 90 in 76fd3a9

out, err = p.communicate()
which means the proxyInfo object on the very next line is not a string anymore:

WMCore/deploy/checkProxy.py

Line 91 in 76fd3a9

proxyInfo = [line for line in out.split('\n') if line]

So all consecutive comparisons/interactions between a str and a bytes type objects fail with the same error (I went further down the code, it happens in few other places as well).

This can explain why @amaltaro suddenly stopped receiving those email warnings regarding proxy expiration for all agents. The script simply never reaches to the point to send an email to anybody, it simply fails with the above error, and because the crontab errors are redirected to the account which is running the crontab, they all go to the cmst1 user e-mail and we never knew. I happened to stumble on those, only because I was fixing the virtenv cronjobs , and those were run with my account, so they started to spam my personal mail. If it was not this coincidence we would have been still let in the dark.

How to reproduce it
Run the cronjob mentioned above manually

Expected behavior
The script for sending those warning e-mails to the relevant person to work.

Additional context and error message
Few more str to bytes comparison errors from the same script:

File /data/WMAgent.venv3/deploy/checkProxy.py:123, in processTimeLeft(sendMail, verbose, proxyInfo, time, mail)
    121 timeLeft = []
    122 for line in proxyInfo:
--> 123     if line.find('timeleft') > -1:
    124         dateReg = re.compile('\d{1,3}[:/]\d{2}[:/]\d{2}')
    125         timeLeft = dateReg.findall(line)[0]

TypeError: argument should be integer or bytes-like object, not 'str'

File /data/WMAgent.venv3/deploy/checkProxy.py:125, in processTimeLeft(sendMail, verbose, proxyInfo, time, mail)
    123 if line.find(b'timeleft') > -1:
    124     dateReg = re.compile('\d{1,3}[:/]\d{2}[:/]\d{2}')
--> 125     timeLeft = dateReg.findall(line)[0]
    126     timeLeft = timeLeft.split(':')[0]
    127     continue

TypeError: cannot use a string pattern on a bytes-like object

File /data/WMAgent.venv3/deploy/checkProxy.py:126, in processTimeLeft(sendMail, verbose, proxyInfo, time, mail)
    124             dateReg = re.compile(b'\d{1,3}[:/]\d{2}[:/]\d{2}')
    125             timeLeft = dateReg.findall(line)[0]
--> 126             timeLeft = timeLeft.split(':')[0]
    127             continue
    128 else:

TypeError: a bytes-like object is required, not 'str'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@todor-ivanov todor-ivanov

Labels
BUG WMAgent
Projects
WMCore quarterly developments
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix str to bytes comparisons in checkProxy.py todor-ivanov/WMCore
1 participant
@todor-ivanov