Add token authentication support #12196

khurtado · 2024-12-05T19:15:33Z

Needed for #12144

Status

tested and working
But external dependency is not completed (condor). Therefore, this functionality is disabled by default.

Description

This does not completely fixes #12144
This is needed to enable token authentication in WMAgent.
Stage-in should be functional with this fix
Stage-out will require changes in the stageout commands
Functionality is disabled by default (until condor setup of tokens is uniform in all schedds)

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

HTCondor token setup
htgettoken (optional) in the CMS runtime image.
Note: The HTCondor token setup is not deployed in all schedds yet.

Additional notes

There is a variable called $BEARER_TOKEN_FILE that if set, HTCondor will write the token there.
This would need to be setup in the host, not the WMAgent container.
This step is however, not critical because this is the reference token.
The actual token that HTCondor transfers comes from

condor_config_val SEC_CREDENTIAL_DIRECTORY_OAUTH

which is in a private system area.
This can be changed in the condor configuration to directly write to /data/certs in the future.

Python bindings
When jobs are submitted via the condor python bindings, inside the WMAgent container, the job is submitted, but /usr/bin/condor_vault_storer does not seem to be triggered.
I am currently working that around by:

Executing a condor_submit with a test job once from the host
Token seems to stay and refresh afterwards
If cms_readonly scope is used, this does not happen and we need a manual refresh, but for production, we don't need any scope.

dmwm-bot · 2024-12-05T19:27:18Z

Jenkins results:

Python3 Unit tests: failed
- 51 new failures
- 26 tests no longer failing
- 6 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/150/artifact/artifacts/PullRequestReport.html

khurtado · 2024-12-05T21:31:56Z

@anpicci @amaltaro FYI

dmwm-bot · 2024-12-06T14:47:45Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/156/artifact/artifacts/PullRequestReport.html

dmwm-bot · 2024-12-06T19:00:56Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/158/artifact/artifacts/PullRequestReport.html

d-ylee · 2024-12-06T19:18:56Z

retest this please

dmwm-bot · 2024-12-06T19:31:57Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/159/artifact/artifacts/PullRequestReport.html

khurtado · 2024-12-09T12:18:40Z

I tested the stage-in with the following workflow:

https://cmsweb-testbed.cern.ch/reqmgr2/fetch?rid=amaltaro_ReReco_Run2022C_LumiMask_khurtado_rereco_lumi_v1_241206_231046_4695

The X509_USER_PROXY environment variable was unset in the job environment for this test.
The condor.out logfile in the jobs show there is a BEARER_TOKEN_FILE but not a X509_USER_PROXY in the environment.

The wmAgent logfiles show the CMSSW run executed successfully and stageOut/dqmUpload failed afterwards.

E.g.:
job1:

2024-12-06 23:40:43,339:INFO:CMSSW:Executing CMSSW. args: ['/bin/bash', '/srv/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh', '', 'el8_amd64_gcc10', 'scramv1', 'CMSSW', 'CMSSW_12_4_14_patch1', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', '']
2024-12-06 23:42:32,781:INFO:PerformanceMonitor:PSS: 2104608; RSS: 2262660; PCPU: 49.5; PMEM: 0.5
2024-12-06 23:47:32,892:INFO:PerformanceMonitor:PSS: 6654177; RSS: 6823764; PCPU: 148; PMEM: 1.7
2024-12-06 23:52:33,083:INFO:PerformanceMonitor:PSS: 6585639; RSS: 6931464; PCPU: 251; PMEM: 1.7
2024-12-06 23:57:33,226:INFO:PerformanceMonitor:PSS: 6569418; RSS: 6919364; PCPU: 292; PMEM: 1.7
2024-12-07 00:02:33,477:INFO:PerformanceMonitor:PSS: 6318371; RSS: 6372732; PCPU: 314; PMEM: 1.6
2024-12-07 00:07:33,577:INFO:PerformanceMonitor:PSS: 6613831; RSS: 6666636; PCPU: 327; PMEM: 1.6
2024-12-07 00:12:33,732:INFO:PerformanceMonitor:PSS: 6645300; RSS: 6699936; PCPU: 337; PMEM: 1.6
2024-12-07 00:17:33,999:INFO:PerformanceMonitor:PSS: 6627041; RSS: 6684564; PCPU: 343; PMEM: 1.6
2024-12-07 00:22:34,221:INFO:PerformanceMonitor:PSS: 6656654; RSS: 6713488; PCPU: 345; PMEM: 1.6
2024-12-07 00:27:34,336:INFO:PerformanceMonitor:PSS: 6628590; RSS: 6707472; PCPU: 349; PMEM: 1.6
2024-12-07 00:27:41,673:INFO:CMSSW:Step cmsRun1: Chirp_WMCore_cmsRun_ExitCode 0
2024-12-07 00:27:41,679:INFO:CMSSW:Step cmsRun1: Chirp_WMCore_cmsRun1_ExitCode 0
2024-12-07 00:27:41,696:INFO:Report:addOutputFile method called with outputModule: AODoutput, aFile: None
2024-12-07 00:27:41,696:INFO:Report:addOutputFile method fileRef: , whole tree: {}
2024-12-07 00:27:41,696:INFO:Report:addOutputFile method called with outputModule: DQMoutput, aFile: None
2024-12-07 00:27:41,697:INFO:Report:addOutputFile method fileRef: , whole tree: {}
2024-12-07 00:27:41,697:INFO:Report:addOutputFile method called with outputModule: MINIAODoutput, aFile: None
2024-12-07 00:27:41,697:INFO:Report:addOutputFile method fileRef: , whole tree: {}
2024-12-07 00:27:44,307:INFO:CMSSW:Steps.Executors.CMSSW.post called
2024-12-07 00:27:44,309:INFO:ExecuteMaster:StepName: cmsRun1, StepType: CMSSW, with result: 0
2024-12-07 00:27:44,366:INFO:Executor:Steps.Executor logging started
2024-12-07 00:27:44,368:INFO:StageOut:Steps.Executors.StageOut.pre called
2024-12-07 00:27:44,368:INFO:StageOut:Steps.Executors.StageOut.execute called
2024-12-07 00:27:44,368:INFO:StageOut:StageOut override is: stageOut1.userDN = None
<snip>
2024-12-07 00:27:44,372:INFO:StageOutMgr:==== Stageout configuration finish ====
2024-12-07 00:27:44,372:INFO:StageOut:Beginning report processing for step cmsRun1
2024-12-07 00:27:44,374:INFO:StageOutMgr:==>Working on file: /store/unmerged/DMWM/MuonEG/AOD/ReReco_Run2022C_LumiMask_khurtado_rereco_lumiv1-v11/00000/1193f448-1f63-4dc6-8e03-f6296c80d06a.root
2024-12-07 00:27:44,374:INFO:StageOutMgr:===> Attempting 2 Stage Outs
2024-12-07 00:27:44,374:INFO:StageOutMgr:LFN to PFN match made:
LFN: /store/unmerged/DMWM/MuonEG/AOD/ReReco_Run2022C_LumiMask_khurtado_rereco_lumiv1-v11/00000/1193f448-1f63-4dc6-8e03-f6296c80d06a.root
PFN: root://cmsdcadisk.fnal.gov//dcache/uscmsdisk/store/unmerged/DMWM/MuonEG/AOD/ReReco_Run2022C_LumiMask_khurtado_rereco_lumiv1-v11/00000/1193f448-1f63-4dc6-8e03-f6296c80d06a.root

2024-12-07 00:27:44,374:INFO:StageOutImpl:Creating output directory...
2024-12-07 00:27:44,376:INFO:StageOutImpl:Running the stage out...
2024-12-07 00:27:44,656:INFO:StageOutImpl:Command exited with status: 151
Output message: stdout: Local File Size is: 550353683
Remote File Size is:
Local File Checksum is: 0e560234
Remote File Checksum is:
ERROR: Size or Checksum Mismatch between local and SE
rm /dcache/uscmsdisk/store/unmerged/DMWM/MuonEG/AOD/ReReco_Run2022C_LumiMask_khurtado_rereco_lumiv1-v11/00000/1193f448-1f63-4dc6-8e03-f6296c80d06a.root : [ERROR] Error response: permission denied

job2:

2024-12-07 03:20:21,870:INFO:CMSSW:RUNNING SCRAM SCRIPTS
2024-12-07 03:20:21,870:INFO:CMSSW:Executing CMSSW. args: ['/bin/bash', '/srv/job/WMTaskSpace/cmsRun1/cmsRun1-main.sh', '', 'el8_amd64_gcc10', 'scramv1', 'CMSSW', 'CMSSW_12_4_14_patch1', 'FrameworkJobReport.xml', 'cmsRun', 'PSet.py', '']
2024-12-07 03:24:36,370:INFO:PerformanceMonitor:PSS: 936741; RSS: 1405664; PCPU: 4.3; PMEM: 0.3
2024-12-07 03:25:20,316:INFO:CMSSW:Step cmsRun1: Chirp_WMCore_cmsRun_ExitCode 0
2024-12-07 03:25:20,321:INFO:CMSSW:Step cmsRun1: Chirp_WMCore_cmsRun1_ExitCode 0
2024-12-07 03:25:20,335:INFO:CMSSW:Steps.Executors.CMSSW.post called
2024-12-07 03:25:20,336:INFO:ExecuteMaster:StepName: cmsRun1, StepType: CMSSW, with result: 0
2024-12-07 03:25:20,345:INFO:Executor:Steps.Executor logging started
2024-12-07 03:25:20,347:INFO:DQMUpload:Steps.Executors.DQMUpload.pre called
2024-12-07 03:25:20,348:INFO:DQMUpload:Steps.Executors.DQMUpload.execute called
2024-12-07 03:25:20,348:INFO:DQMUpload:Beginning report processing for step cmsRun1
2024-12-07 03:25:20,469:INFO:DQMUpload:HTTP Upload is about to start:
 => URL: https://cmsweb-testbed.cern.ch/dqm/dev
 => Filename: /srv/job/WMTaskSpace/cmsRun1/DQM_V0001_R000355870__MuonEG__DMWM-ReReco_Run2022C_LumiMask_khurtado_rereco_lumiv1-v11__DQMIO.root

amaltaro

Kenyi, Andrea, it is not clear to me if this is a test and/or development in progress or not. I suspect you are still working on this, so I am labeling this according to make it clear and avoid mistakes (please remove the label once it's ready for a final review/merge).

In addition, please let me know if you wanted me to look into anything specific. I do not have anything else to add here and changes are looking alright.

etc/submit_py3.sh

khurtado · 2024-12-16T13:37:50Z

@amaltaro This has been tested for stage-in and it is working. However, it depends on condor to be properly setup, otherwise condor will still submit but they will fail with:

 ID      OWNER          HELD_SINCE  HOLD_REASON
1751.0   cmst1          12/16 14:30 Job credentials are not available

Therefore, code review is fine, but we cannot merge until the condor setup is fully defined (as far as I know, there is some automation related development going on regarding this) and deployed in all condor schedds.

dmwm-bot · 2024-12-16T13:46:51Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/215/artifact/artifacts/PullRequestReport.html

dmwm-bot · 2024-12-16T13:55:27Z

Jenkins results:

Python3 Unit tests: succeeded
- 3 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 47 comments to review
Pycodestyle check: succeeded
- 13 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/216/artifact/artifacts/PullRequestReport.html

mapellidario · 2025-01-06T15:11:42Z

I opened a ticket with SI to ask them to install on our schedds what we need to move forward [1]

[1] https://its.cern.ch/jira/projects/CMSSI/issues/CMSSI-122

anpicci · 2025-01-07T09:12:24Z

Thank you @mapellidario !

dmwm-bot · 2025-02-19T18:58:16Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 3 changes in unstable tests
Python3 Pylint check: failed
- 1 warnings and errors that must be fixed
- 4 warnings
- 54 comments to review
Pycodestyle check: succeeded
- 14 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/375/artifact/artifacts/PullRequestReport.html

dmwm-bot · 2025-02-20T20:16:37Z

Jenkins results:

Python3 Unit tests: failed
- 8 new failures
- 4 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 54 comments to review
Pycodestyle check: succeeded
- 14 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/378/artifact/artifacts/PullRequestReport.html

khurtado · 2025-02-20T21:50:51Z

Update:

Token support is disabled by default. When enabled, it shows the following in the main log:

Content under _CONDOR_CREDS: /srv/.condor_creds
total 4
-rw------- 1 cmsgli us_cms 867 Feb 20 21:45 cms.use
CMS token found, setting BEARER_TOKEN_FILE=/srv/.condor_creds/cms.use
======== WMAgent Unpack the job starting at Thu Feb 20 21:45:58 GMT 2025 ========
======== WMAgent Unpack the job finished at Thu Feb 20 21:45:59 GMT 2025 ========

khurtado · 2025-02-20T22:10:09Z

@d-ylee Can you remind me where and when the WMAgent config is copied from in the jenkins setup?

I added one line in the config.
In the manage script:

https://github.com/dmwm/CMSKubernetes/blob/f9002e0c55e953627d8dc20536d27795e67bfd60/docker/pypi/wmagent/bin/manage#L80

This comes from: /data/srv/wmagent/current/etc/WMAgentConfig.py and copied into: /data/srv/wmagent/current/config.py

In Jenkins, I see we create the config directory at some point:

https://github.com/dmwm/WMCore-Jenkins/blob/ddc6ac31be6f248013ddf2143a7c7cca1a39b1c2/WMCore-Test-Base/docker-compose.yml#L117

but I'm not sure if we pick the config from WMA_TAG, or if we override this config with the one from the PR into /data/srv/wmagent/current/etc/WMAgentConfig.py

In short, do you remember if we propagate changes from the etc directory through the jenkins setup? or do we use the etc from the original WMA_TAG without modifications?

dmwm-bot · 2025-02-20T22:37:02Z

Jenkins results:

Python3 Unit tests: succeeded
- 1 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 54 comments to review
Pycodestyle check: succeeded
- 14 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/380/artifact/artifacts/PullRequestReport.html

khurtado · 2025-02-20T22:49:35Z

@amaltaro @anpicci I have added an option to disable usage of tokens by default. With this, we can safely merge this PR without breaking agents sitting on nodes without the proper token setup.

anpicci

@khurtado, sounds great, thank you! May you report also here that your tests were successful?

khurtado · 2025-02-24T22:29:51Z

@anpicci Yes!

The option in the agent to enable/disable the token functionality is working as expected. I tested with a couple of worklows in testbed.

amaltaro

@khurtado thank you for providing these changes. They are looking good to me, however can you please:

if you agree, update that 1 line comment I made
squash these commits accordingly
?

Thanks!

amaltaro · 2025-02-26T16:15:24Z

etc/submit_py3.sh

+        # exit 1106
+    fi
+else
+    echo "No condor auth/token credentials directory found."


How about being more explicit and saying something like "Variable _CONDOR_CREDS is not defined and no condor auth..."?

…n by default.

khurtado · 2025-02-26T18:20:44Z

@khurtado thank you for providing these changes. They are looking good to me, however can you please:

if you agree, update that 1 line comment I made

squash these commits accordingly
?

Thanks!

Sounds good, I just updated that line to be more explicit and squashed the commits.

dmwm-bot · 2025-02-26T18:29:48Z

Jenkins results:

Python3 Unit tests: succeeded
- 4 changes in unstable tests
Python3 Pylint check: succeeded
- 4 warnings
- 54 comments to review
Pycodestyle check: succeeded
- 14 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/399/artifact/artifacts/PullRequestReport.html

amaltaro

Great, thanks Kenyi. It all looks good now.

khurtado requested review from amaltaro and anpicci December 9, 2024 12:18

khurtado mentioned this pull request Dec 9, 2024

Adopt token for WMAgent, test stage-in #12199

Closed

amaltaro reviewed Dec 13, 2024

View reviewed changes

etc/submit_py3.sh Outdated Show resolved Hide resolved

amaltaro added the PR: Work in progress label Dec 13, 2024

khurtado added PR: Do not merge yet and removed PR: Work in progress labels Dec 16, 2024

khurtado force-pushed the tokens1 branch from 23b52aa to f53a5b0 Compare December 16, 2024 13:45

anpicci approved these changes Jan 4, 2025

View reviewed changes

khurtado force-pushed the tokens1 branch from 781c358 to 67d1d3a Compare February 20, 2025 20:06

khurtado force-pushed the tokens1 branch from 67d1d3a to 44cc7ae Compare February 20, 2025 22:23

khurtado requested review from anpicci and amaltaro February 20, 2025 22:48

khurtado removed the PR: Do not merge yet label Feb 21, 2025

anpicci approved these changes Feb 24, 2025

View reviewed changes

amaltaro requested changes Feb 26, 2025

View reviewed changes

khurtado force-pushed the tokens1 branch from 44cc7ae to 3d21c55 Compare February 26, 2025 18:17

Add token authentication support, but disable cms token authenticatio…

7d1d3c7

…n by default.

khurtado force-pushed the tokens1 branch 2 times, most recently from 3efbb9f to 7d1d3c7 Compare February 26, 2025 18:19

amaltaro self-requested a review February 26, 2025 20:59

amaltaro approved these changes Feb 26, 2025

View reviewed changes

amaltaro merged commit 45cf495 into dmwm:master Feb 26, 2025
3 checks passed

amaltaro mentioned this pull request Feb 27, 2025

Adopt token for WMAgent stage-out #12144

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add token authentication support #12196

Add token authentication support #12196

khurtado commented Dec 5, 2024 •

edited

Loading

dmwm-bot commented Dec 5, 2024

khurtado commented Dec 5, 2024

dmwm-bot commented Dec 6, 2024

dmwm-bot commented Dec 6, 2024

d-ylee commented Dec 6, 2024

dmwm-bot commented Dec 6, 2024

khurtado commented Dec 9, 2024

amaltaro left a comment

khurtado commented Dec 16, 2024 •

edited

Loading

dmwm-bot commented Dec 16, 2024

dmwm-bot commented Dec 16, 2024

mapellidario commented Jan 6, 2025

anpicci commented Jan 7, 2025

dmwm-bot commented Feb 19, 2025

dmwm-bot commented Feb 20, 2025

khurtado commented Feb 20, 2025

khurtado commented Feb 20, 2025 •

edited

Loading

dmwm-bot commented Feb 20, 2025

khurtado commented Feb 20, 2025

anpicci left a comment

khurtado commented Feb 24, 2025 •

edited

Loading

amaltaro left a comment

amaltaro Feb 26, 2025

khurtado Feb 26, 2025

khurtado commented Feb 26, 2025

dmwm-bot commented Feb 26, 2025

amaltaro left a comment

Add token authentication support #12196

Add token authentication support #12196

Conversation

khurtado commented Dec 5, 2024 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

Additional notes

dmwm-bot commented Dec 5, 2024

khurtado commented Dec 5, 2024

dmwm-bot commented Dec 6, 2024

dmwm-bot commented Dec 6, 2024

d-ylee commented Dec 6, 2024

dmwm-bot commented Dec 6, 2024

khurtado commented Dec 9, 2024

amaltaro left a comment

Choose a reason for hiding this comment

khurtado commented Dec 16, 2024 • edited Loading

dmwm-bot commented Dec 16, 2024

dmwm-bot commented Dec 16, 2024

mapellidario commented Jan 6, 2025

anpicci commented Jan 7, 2025

dmwm-bot commented Feb 19, 2025

dmwm-bot commented Feb 20, 2025

khurtado commented Feb 20, 2025

khurtado commented Feb 20, 2025 • edited Loading

dmwm-bot commented Feb 20, 2025

khurtado commented Feb 20, 2025

anpicci left a comment

Choose a reason for hiding this comment

khurtado commented Feb 24, 2025 • edited Loading

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Feb 26, 2025

Choose a reason for hiding this comment

khurtado Feb 26, 2025

Choose a reason for hiding this comment

khurtado commented Feb 26, 2025

dmwm-bot commented Feb 26, 2025

amaltaro left a comment

Choose a reason for hiding this comment

khurtado commented Dec 5, 2024 •

edited

Loading

khurtado commented Dec 16, 2024 •

edited

Loading

khurtado commented Feb 20, 2025 •

edited

Loading

khurtado commented Feb 24, 2025 •

edited

Loading