Releases: dmwm/WMCore
Releases · dmwm/WMCore
WMCore 2.1.4 central services and WMAgent production release
This release provides many important changes, such as: support to compressed (gzip) response data (clients are invited to update their tools), MSOutput service migrated from localhost to MongoDB as A Service, microservices are packaged separately and there is GH action for building and uploading all the WMCore services to PyPi.
It also brings in some bug-fixes and enhancements in multiple WMCore services.
Release date: 7 November 2022.
Changes since release: 2.1.3.
Central services
Software stack
- Update dbs3-client version from 4.0.10 to 4.0.11 (Alan Malta Rodrigues) #11334
Features and/or feature changes
- Split microservices into separate packages (Erik Gough) #11324
- Build WMCore services and publish them to pypy, using tag pushes (Kenyi Hurtado) #11318 #11337 #11355
- Add support to gzip response at the REST layer (Alan Malta Rodrigues) #11343
- Disable automatic data decompression in libcurl - ACCEPT_ENCODING (Alan Malta Rodrigues) #11343
- Switch MSOutput to use MongoDBAAS. (Todor Ivanov) #11280
Bug Fixes
- use is_alive with thread objs (isAlive deprecated) (Benjamin Tovar) #11323
- MSOutput: use AlertManagerAPI from MSCore (#11357) (Alan Malta Rodrigues)
Enhancements
WMAgent
Features and/or feature changes
- Enable compression of block data injected into DBSServer (Alan Malta Rodrigues) #11325
- add WebDav way to say "permission denied" to gfal_copy add webdav Insufficient Storage StageOut error (Stefano Belforte) #11328
- Trap DBS3Upload exceptions when fetching child/parent info from wmstats (Alan Malta Rodrigues) #11352
- Fix T0 updaterequest update function to avoid same status transition (Alan Malta Rodrigues) #11353
- Add support to gzip response at the REST layer (Alan Malta Rodrigues) #11343
- Disable automatic data decompression in libcurl - ACCEPT_ENCODING (Alan Malta Rodrigues) #11343
Bug Fixes
- Set the database dialect from the database URL && Make socket parameter optional. (Todor Ivanov) #11348
Enhancements
- Scram.py - decode Popen stdout and stderr (Dario Mapelli) #11321
WMCore 2.1.3 central services production release
This release contains a feature change for MSRuleCleaner pileup alerts, it brings in a new version of CMSMonitoring with a fix for stomp heartbeat.
On the agent side, it has a few bugfixes and a new condor job classad for the job required architecture.
Finally, it has some other enhancements and documentation contributions.
Release date: 10 October 2022.
Changes since release: 2.1.2.
Central services
Software stack
Features and/or feature changes
- Skip alert for RelVals with deletable pileup (Alan Malta Rodrigues) #11283
Bug Fixes
Enhancements
- Update schematics for WM, MSTransferor and MSMonitor (Alan Malta Rodrigues) #11294
WMAgent
Features and/or feature changes
- Create job classad for REQUIRED_ARCH (Alan Malta Rodrigues) #11298
Bug Fixes
- Convert bin/workflowCompletion.py to python3-only; fix url bug (Alan Malta Rodrigues) #11286
- Fix python version for patchComponent.sh (#11272) (Todor Ivanov) #11272
Enhancements
WMCore 2.1.2 central services and WMAgent production release
This release contains database and workflow spec schema changes, with the removal of some legacy parameters from the PhEDEx era.
It also brings in many python3 third-party libraries updates, including for CMSMonitoring and dbs3-client.
Last but not least, it contains continuous bugfixes and enhancements in multiple WMCore services.
Release date: 7 September 2022.
Changes since release: 2.1.1.
Central services
Software stack
- Update CMSMonitoring python library to 0.6.7 (Alan Malta Rodrigues) #11255
- Update WMCore dependencies (Valentin Kuznetsov) #11258
- New version of DBS3Client 4.0.10 (Valentin Kuznetsov) #11261
Features and/or feature changes
- Validate StdBase ValidStatus parameter for DBS output data (Alan Malta Rodrigues) #11237
- Dropping PhEDEx args (Jhonatan Amado) #11181
- [MSRuleCleaner] Create alert for pileup containers eligible for rule deletion (Alan Malta Rodrigues) #11264
- Remove extra request parameter checks for [Non]CustodialSubType && AutoApproveSubscriptionSites. (Todor Ivanov) #11270
Bug Fixes
- Fix strToBool validate function to fail on integer values (Alan Malta Rodrigues) #11251
Enhancements
WMAgent
Features and/or feature changes
- Use DBS ValidStatus from the spec for DBS3 data injection (Alan Malta Rodrigues) #11237
- Dropping PhEDEx args (Jhonatan Amado) #11181
Bug Fixes
- Combines block records in single dict that includes all associated wfs (#11245) (German Felipe Giraldo Villa) #11245
- Remove threadlocal parameter for SQLAlchemy as it is deprecated in new release (Valentin Kuznetsov) #11257
- Cast SQLAlchemy RMKeyView object to list (Alan Malta Rodrigues) #11268
Enhancements
WMCore 2.1.1 central services production release
This release brings in a backward-compatible authorization layer for ReqMgr2. It also defines Nano workflows to use a Dataset level granularity (before it was set to Block). Input data placement for Nano workflows will also be done at dataset level and 2 copies will be request.
This version also comes with the very latest stable CouchDB release.
Release date: 5 August 2022.
Changes since release: 2.1.0.
Central services
Software stack
- Update Py3 CMSMonitoring to 0.6.5 (Alan Malta Rodrigues) cms-sw/cmsdist#8002
- Updated CouchDB to version 3.2.2 (Alan Malta Rodrigues) cms-sw/cmsdist#8001
Features and/or feature changes
- Adopt ReqMgr2 authorization based on the request status (Alan Malta Rodrigues) #11223
- Implement support for gzipped stream for RucioConMon (Valentin Kuznetsov) #11142
- Move some Rucio utils over to RucioUtils module (Alan Malta Rodrigues) #11218
- Set start policy to Dataset for workflows processing MINIAODSIM dataset (Alan Malta Rodrigues) #11219
- MS: Implement new object for Nano input data placement (Alan Malta Rodrigues) #11219
Bug Fixes
Enhancements
- Update CMSMonitoring version in the requirements file to 0.6.5 (Alan Malta Rodrigues) #11229
- Update DBS mock calls and data with MINIAODSIM (Alan Malta Rodrigues) #11219
WMAgent
Features and/or feature changes
- Allow list of multiple architectures (khurtado) #11228
Bug Fixes
Enhancements
- Add ExecuteDAO script. (Todor Ivanov) #11174
WMCore 2.1.0 central services production release
This release was focused on the migration from CouchDB 1.6 to CouchDB 3+, making the WMCore framework fully compatible with CouchDB 3+. It brings in a few bug fixes and continuous enhancements for the system.
Release date: 13 July 2022.
Changes since release: 2.0.4.
Central services
Software stack
- Update stomp.py package from 4.1.21 to 7.0.0 - required for CMSMonitoring (Alan Malta Rodrigues) cms-sw/cmsdist#7984
- Update Py3 CMSMonitoring to 0.6.4 (Alan Malta Rodrigues) cms-sw/cmsdist#7983
- Fix py3-pyjwt PyPi name (Alan Malta Rodrigues) cms-sw/cmsdist#7929
- New specs/version for Erlang and CouchDB 3.1.2 (Alan Malta Rodrigues) cms-sw/cmsdist#7218
- Many deployment configuration changes like (Alan Malta Rodrigues) dmwm/deployment#1154 , 1157, 1158, 1159 and 1162
Features and/or feature changes
- Remove all the CMS authz logic from the CouchDB validate functions (Alan Malta Rodrigues) #11191
- Bundle of fixes required for migration to CouchDB 3.x (Alan Malta Rodrigues) #11001
Bug Fixes
- Consistent return data type for closing workqueue elements (Alan Malta Rodrigues) #11193
Enhancements
- Create filter function to drop deleted documents (Alan Malta Rodrigues) #11196
- Give enough time for replication to complete in unit tests (Alan Malta Rodrigues) #11185
WMAgent
Features and/or feature changes
- Provide uniform way to handle DBS server errors (Valentin Kuznetsov) #11176
- Bundle of fixes required for migration to CouchDB 3.x (Alan Malta Rodrigues) #11001
Bug Fixes
- Load credentials when saving spec file in local workqueue (Alan Malta Rodrigues) #11204
- Ensure dataset_lifetime bind is defined for DBS3Buffer/NewSubscription (Alan Malta Rodrigues) #11179
Enhancements
WMAgent production patch release: 2.0.4.patch1
This is a major release upgrade because it's the first WMAgent stable release compatible with CouchDB 3.1 (and dependent on). It also has a few important features and bug-fixes required for data taking and the T0 agent. Lastly, it has the usual and continuous code and stability improvements.
Release date: 16 June 2022.
WMAgent changes since release: 2.0.2
WMAgent
Software stack
- Upgrade CouchDB to 3.1.2, Erlang to 22.3.4.19 (Alan Malta Rodrigues) cms-sw/cmsdist#7218
- Bring dbs3-client version to 4.0.8 (Alan Malta Rodrigues) #11138
- Removed pep8; added pycodestyle 2.8.0; updated pylint to 2.13.5 (Alan Malta Rodrigues) cms-sw/cmsdist#7846
- Update PyJWT from version 2.3.0 to 2.4.0 (snyk-bot) #11155
Features and/or feature changes
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11083
- If job requires rhel8 container, use slc7 python libraries (Alan Malta Rodrigues) #11077
- Support adding diskless resources into the database (Alan Malta Rodrigues) #11097
- Remove PFNs from a final document we send to WMArchive (Valentin Kuznetsov) #10998
- change fallback for missing system xrootd-client (Dirk Hufnagel) #11117
- Remove support for Unpacking user tarballs. No longer used by CRAB. (khurtado) #11114
- Enforce dataset_lifetime column to be integer and not null (Alan Malta Rodrigues) #11115
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
- Split Deletable Blocks checks in two stages. (Todor Ivanov) #11127
- New CouchDB 3.x APIs for tracking replications; fix replication check in AgentStatusPoller (Alan Malta Rodrigues) #11039
- Add extra protection for T0 to prevent archival of workflows having blocks not yet deleted. (Todor Ivanov) #11169
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11067
- [Tier0] Check CMSSW version before getting sim-datatier map (germanfgv) #11110
- Check if key exists when deleting standalone documents in Couch (Alan Malta Rodrigues) #11165
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11088
- [PY2] Remove cPickle support (Erik Gough) #11096
- Update wmagent deploy script to add opportunistic resources one by one (Alan Malta Rodrigues) #11097
- Minor test json template updates (Alan Malta Rodrigues) #10864
- Add T3_US_Lancium. (Todor Ivanov) #11126
- Add buildAllDep option && Fix package verstion estimation for ParseSpecCmsswdist. (Todor Ivanov) #11132
WMCore 2.0.4 central services and WMAgent production release
This release brings in a policy change for pileup input data placement in MSTransferor. It also brings in most of the required changes to support the newest version of CouchDB. Finally, it comes with usual enhancements to the code and documentation.
Release date: 9 June 2022.
Changes since release: 2.0.3.
Central services
Software stack
- Update PyJWT from version 2.3.0 to 2.4.0 (snyk-bot) #11155
Features and/or feature changes
- [MSTransferor] Create rule for each pileup location (Alan Malta Rodrigues) #11143 #11153
- Remove CouchDB all_or_nothing option for bulk API (Alan Malta Rodrigues) #11011
- Remove CouchDB JobDump/highestJobID view (Alan Malta Rodrigues) #11161
Bug Fixes
- [Microservices] Fix logic to check for empty results in MS/Common (Alan Malta Rodrigues) #11148
- Avoid raising exception when ReqMgr2 request is not found (Alan Malta Rodrigues) #11159
Enhancements
- [MSTransferor] Reorganization and classification of workflow templates and placement policies (Alan Malta Rodrigues) #11143
WMAgent
Features and/or feature changes
- Split Deletable Blocks checks in two stages. (Todor Ivanov) #11127
- New CouchDB 3.x APIs for tracking replications; fix replication check in AgentStatusPoller (Alan Malta Rodrigues) #11039
- Add extra protection for T0 to prevent archival of workflows having blocks not yet deleted. (Todor Ivanov) #11169
Bug Fixes
- Check if key exists when deleting standalone documents in Couch (Alan Malta Rodrigues) #11165
Enhancements
WMCore 2.0.3 central services production release
This release brings in many awesome features, such as: some initial support to IAM tokens; full support to workflows with growing input dataset; new policy for Release Validation output data placement, with a distinction of pre-releases from the other samples.
It also includes a few bug fixes required for the DBSServer migration to the Go implementation. Finally, it comes with many enhancements to the code and documentation.
Release date: 7 April 2022.
Changes since release: 2.0.2.
Central services
Software stack
- Bring dbs3-client version to 4.0.8 (Alan Malta Rodrigues) #11138
- Removed pep8; added pycodestyle 2.8.0; updated pylint to 2.13.5 (Alan Malta Rodrigues) cms-sw/cmsdist#7846
Features and/or feature changes
- Add bin and dependencies to wmcore PyPI package (Erik Gough) #11103
- Remove block open logic from DBS3Reader and WorkQueue (Alan Malta Rodrigues) #11123
- Properly handle OpenRunningTimeout in WorkQueue (Alan Malta Rodrigues) #11119 #11135
- Initial script to create IAM token (Valentin Kuznetsov) #11093 #11134
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- [MSTransferor] Lock the whole input container for growing workflows (Alan Malta Rodrigues) #11141
- [MSOutput] Apply different rule lifetime for RelVal pre-release samples (Alan Malta Rodrigues) #11139
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
- Add support to OpenRunningTimeout for TaskChain/StepChain (Alan Malta Rodrigues) #11135
Bug Fixes
- Update ReqMgr2 CP thread to properly deal with aborted/force-complete workflows (Alan Malta Rodrigues) #11113
Enhancements
- move ckey/cert functions to Utils.CertTools (Valentin Kuznetsov) #11101
- Remove no longer needed code in the StatusChangeTasks ReqMgr2 thread (Alan Malta Rodrigues) #11095
- Create script to cancel GQEs (Alan Malta Rodrigues) #11113
- Switch from pep8 to pycodestyle (Erik Gough) #11128
WMAgent
Features and/or feature changes
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11083
- If job requires rhel8 container, use slc7 python libraries (Alan Malta Rodrigues) #11077
- Support adding diskless resources into the database (Alan Malta Rodrigues) #11097
- Remove PFNs from a final document we send to WMArchive (Valentin Kuznetsov) #10998
- change fallback for missing system xrootd-client (Dirk Hufnagel) #11117
- Remove support for Unpacking user tarballs. No longer used by CRAB. (khurtado) #11114
- Enforce dataset_lifetime column to be integer and not null (Alan Malta Rodrigues) #11115
- speed up listDatasetFileDetails API (Valentin Kuznetsov) #11099
- Add support for IAM token in the underlying pycurl_manager module (Valentin Kuznetsov) #10939
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11067
- [Tier0] Check CMSSW version before getting sim-datatier map (germanfgv) #11110
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11088
- [PY2] Remove cPickle support (Erik Gough) #11096
- Update wmagent deploy script to add opportunistic resources one by one (Alan Malta Rodrigues) #11097
- Minor test json template updates (Alan Malta Rodrigues) #10864
- Add T3_US_Lancium. (Todor Ivanov) #11126
- Add buildAllDep option && Fix package verstion estimation for ParseSpecCmsswdist. (Todor Ivanov) #11132
WMAgent patch release: 2.0.2.patch1
Release date: 12 April 2022.
Changes since release: 2.0.2.
In addition to the release notes provided under within the tag 2.0.2, this patch release brings in the following.
WMAgent
Features and/or feature changes
- Default job wrapper to use rhel7 python libraries, otherwise use the same as CMSSW (Alan Malta Rodrigues) #11090
Bug Fixes
- Assign the correct activity for Tape_Test RSEs (Jhonatan Amado) #11090
- When ScramArch is empty str/list/None, return any as required_os (Alan Malta Rodrigues) #11090
Enhancements
- Enhance logic to map ScramArch to OS (Alan Malta Rodrigues) #11090
WMCore 2.0.2 central services and WMAgent production release
This release has support to compressed data in HTTP requests, relying on the automatic
decoding algorithms built with libcurl. For WMAgent, it has a database schema change for
the processing version; and it has full support to EL8 workflows.
Other than that, many enhancements to the WMCore workflow, documentation and overall
improvements have been made.
Release date: 7 April 2022.
Changes since release: 2.0.1.
WMAgent changes since release: 1.5.7
Central services
Software stack
Features and/or feature changes
- Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
- Add usage of gzip encoding (Valentin Kuznetsov) #11056
- Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
- [MSUnmerged] Fetch protectedLFNs for T0 (Todor Ivanov) #11050
Bug Fixes
- [MSTransferor] do not bypass Pileup rules in INJECT/REPLICATING state (Alan Malta Rodrigues) #11061
Enhancements
- Document the use of docstrings and best practices in the project (Alan Malta Rodrigues) #11052
- Document Services cache deletion on exit (Alan Malta Rodrigues) #10915
WMAgent
Features and/or feature changes
- Define gzip encoding by default in the headers; properly decompress it (Valentin Kuznetsov) #11041
- Adjust processing version to be integer (Valentin Kuznetsov) #11043 #11059
- [CouchDB] Do not sanitize couch url at the component layer (Alan Malta Rodrigues) #11045
- Add el8 flavor to the list of rhel8 container jobs (Alan Malta Rodrigues) #11060 #11062
- Add usage of gzip encoding (Valentin Kuznetsov) #11056
- Use Content-Encoding to automatically decode body in RequestHandler (Alan Malta Rodrigues) #11069 #11070
- Add parameters to rucio datasets in RucioInjector (Jhonatan Amado) #10989
- Add lifetime to RucioContainers shipping to disk sites (Jhonatan Amado) #10950
- make sure proxy is 2048 bit. Fix #11005 (belforte) #11006
- Explicitly create proxies with 2048 bits (Alan Malta Rodrigues) #11008
Bug Fixes
- Prevent excessive calls for agent config via ReqMgrAux (Erik Gough) #11033
- Fix a few DBS3Reader APIs, likely unused ones (Alan Malta Rodrigues) #10990
- DBS3Upload: check error message from the response body attribute (Alan Malta Rodrigues) #10961
Enhancements
- Better logging for DBS block insertion in DBS3Upload (Alan Malta Rodrigues) #11058
- Download script and insert US opportunistic resources during deployment (Alan Malta Rodrigues) #11047
- Update mocked DBS APIs and data (Alan Malta Rodrigues) #10990
- Enhancements to the GH issues and PRs workflows (Alan Malta Rodrigues) #11021
- Update deploy-wmagent for 1.5.7.patch1; remove deprecated py2 option (Alan Malta Rodrigues) #10966
- Update error message for jobs killed from site state transition (Alan Malta Rodrigues) #10958