Move filter replication json document under etc/ #12150

amaltaro · 2024-10-17T18:53:13Z

Status

ready

Description

Given that the json file isn't packaged into the distribution uploaded to PyPi, nor it can be packaged by instrumenting setup_dependencies.py [1] and [2]; I decided to move the json file under the etc/ directory, which can be looked up through the WMA_DEPLOY_DIR environment variable (suffixed with etc).

Summary of changes are:

remove ParentQueueUrl from the replication filter function
reposition replication_selector.json under etc/
fixed AgentStatusPoller unit tests have been instrumented such that it works in all scenarios (CC7 RPM, CC7 Docker, Alma9).
lastly, added RucioUtils_t.RucioUtilsTest:testWeightedChoice to the unstable tests.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

Follow up of #12143

External dependencies / deployment changes

[1] ### Using the statics option like:

        'statics': ['bin+',
                         ...
                         'src/python/WMComponent/AgentStatusWatcher/replication_selector.json'],

### python3 setup.py clean sdist, fails with:
Traceback (most recent call last):
  File "/home/alan/build_pypi/WMCore/setup.py", line 26, in <module>
    data_files = list_static_files(dependencies[package_name])
  File "/home/alan/build_pypi/WMCore/setup_build.py", line 118, in list_static_files
    static_files.extend(data_files_for(static_dir))
  File "/home/alan/build_pypi/WMCore/setup_build.py", line 96, in data_files_for
    for ifile in os.listdir(dir):
NotADirectoryError: [Errno 20] Not a directory: 'src/python/WMComponent/AgentStatusWatcher/replication_selector.json'

[2] ### Using the modules option like:

        'modules': ['WMComponent.AgentStatusWatcher.replication_selector.json'],

### python3 setup.py clean sdist, fails with:
writing manifest file 'src/python/wmagent.egg-info/SOURCES.txt'
error: package directory 'src/python/WMComponent/AgentStatusWatcher/replication_selector' does not exist

amaltaro · 2024-10-17T19:01:42Z

And I can confirm that:

the json file is now packaged into the tarball uploaded to PyPi (tested private building in my private VM)
and upon patching AgentStatusWatcher - and locating this file under /usr/local/etc - it is able to find it and properly create the database replication documents.

cmsdmwmbot · 2024-10-17T19:03:28Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 27 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15336/artifact/artifacts/PullRequestReport.html

todor-ivanov

looking good. Thanks @amaltaro

mapellidario

thanks alan! looks perfect

amaltaro · 2024-10-18T11:00:09Z

Thanks for the review!
I still have to figure what to do with the failing unit test though:

    WMComponent_t.AgentStatusWatcher_t.AgentStatusWatcher_t.AgentStatusWatcher_t:testAgentStatusPoller changed from success to error

given that Jenkins is currently running in the RPM base deployment, I will have to adapt it somehow.

Do not set ParentQueueUrl in the selector filter

cmsdmwmbot · 2024-10-18T18:00:25Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 1 tests no longer failing
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 27 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15337/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2024-10-18T21:53:11Z

Can one of the admins verify this patch?

amaltaro · 2024-10-19T03:38:17Z

test this please

cmsdmwmbot · 2024-10-19T04:02:24Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 1 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 27 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 5 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15341/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2024-10-20T15:16:18Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 2 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 32 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 7 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15343/artifact/artifacts/PullRequestReport.html

cmsdmwmbot · 2024-10-21T19:12:17Z

Jenkins results:

Python3 Unit tests: failed
- 2 new failures
- 1 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 32 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 8 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15348/artifact/artifacts/PullRequestReport.html

amaltaro · 2024-10-21T19:56:11Z

test this please

cmsdmwmbot · 2024-10-21T20:11:32Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 tests no longer failing
- 2 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 32 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 9 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15350/artifact/artifacts/PullRequestReport.html

unit test with all possible fallbacks unit test: WMCORE_ROOT has double install dir now

amaltaro · 2024-10-21T21:39:17Z

test this please

cmsdmwmbot · 2024-10-21T22:13:49Z

Jenkins results:

Python3 Unit tests: failed
- 1 new failures
- 1 tests no longer failing
- 1 changes in unstable tests
Python3 Pylint check: failed
- 5 warnings and errors that must be fixed
- 3 warnings
- 32 comments to review
Pylint py3k check: succeeded
Pycodestyle check: succeeded
- 9 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15352/artifact/artifacts/PullRequestReport.html

mapellidario

thanks for the changes and upgrading the list of unstable unittests!

amaltaro · 2024-10-22T11:23:14Z

Thanks Dario! Failing unit test seems to actually be unstable:

    WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProductionMultiQueue changed from success to failure

let us keep an eye on it in near future PRs and set it appropriately, if needed.

amaltaro requested review from mapellidario and todor-ivanov October 17, 2024 19:01

todor-ivanov approved these changes Oct 18, 2024

View reviewed changes

mapellidario approved these changes Oct 18, 2024

View reviewed changes

amaltaro added 2 commits October 18, 2024 13:10

Move filter replication json document under etc/

a23be0c

Do not set ParentQueueUrl in the selector filter

Update workqueue selector filter to no longer use ParentQueueUrl

62aab36

amaltaro force-pushed the fix-11192-take2 branch from 8c68a0d to 62aab36 Compare October 18, 2024 17:13

amaltaro added 2 commits October 21, 2024 16:15

fix unit test for RPM based

e55bc90

unit test with all possible fallbacks unit test: WMCORE_ROOT has double install dir now

add RucioUtils_t.RucioUtilsTest:testWeightedChoice to the unstable test

ec125dd

amaltaro force-pushed the fix-11192-take2 branch from 789459a to ec125dd Compare October 21, 2024 20:15

amaltaro requested a review from vkuznet October 21, 2024 20:19

amaltaro requested a review from mapellidario October 22, 2024 03:19

mapellidario approved these changes Oct 22, 2024

View reviewed changes

amaltaro merged commit 3917282 into dmwm:master Oct 22, 2024
2 of 4 checks passed

amaltaro added the PR: Need wmagent branch label Oct 22, 2024

amaltaro mentioned this pull request Oct 22, 2024

Move filter replication json document under etc/ - wmagent branch #12154

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move filter replication json document under etc/ #12150

Move filter replication json document under etc/ #12150

amaltaro commented Oct 17, 2024 •

edited

Loading

amaltaro commented Oct 17, 2024

cmsdmwmbot commented Oct 17, 2024

todor-ivanov left a comment

mapellidario left a comment

amaltaro commented Oct 18, 2024

cmsdmwmbot commented Oct 18, 2024

cmsdmwmbot commented Oct 18, 2024

amaltaro commented Oct 19, 2024

cmsdmwmbot commented Oct 19, 2024

cmsdmwmbot commented Oct 20, 2024

cmsdmwmbot commented Oct 21, 2024

amaltaro commented Oct 21, 2024

cmsdmwmbot commented Oct 21, 2024

amaltaro commented Oct 21, 2024

cmsdmwmbot commented Oct 21, 2024

mapellidario left a comment

amaltaro commented Oct 22, 2024

Move filter replication json document under etc/ #12150

Move filter replication json document under etc/ #12150

Conversation

amaltaro commented Oct 17, 2024 • edited Loading

Status

Description

Is it backward compatible (if not, which system it affects?)

Related PRs

External dependencies / deployment changes

amaltaro commented Oct 17, 2024

cmsdmwmbot commented Oct 17, 2024

todor-ivanov left a comment

Choose a reason for hiding this comment

mapellidario left a comment

Choose a reason for hiding this comment

amaltaro commented Oct 18, 2024

cmsdmwmbot commented Oct 18, 2024

cmsdmwmbot commented Oct 18, 2024

amaltaro commented Oct 19, 2024

cmsdmwmbot commented Oct 19, 2024

cmsdmwmbot commented Oct 20, 2024

cmsdmwmbot commented Oct 21, 2024

amaltaro commented Oct 21, 2024

cmsdmwmbot commented Oct 21, 2024

amaltaro commented Oct 21, 2024

cmsdmwmbot commented Oct 21, 2024

mapellidario left a comment

Choose a reason for hiding this comment

amaltaro commented Oct 22, 2024

amaltaro commented Oct 17, 2024 •

edited

Loading