v3.00.57 - closer to fine

Summary

"The less I seek my source for some definitive, the closer I am to fine." -- Indigo Girls.

The last version (3.0.56) was the first one where we thought sr3 could replace any
v2 configuration. As we did so, a few new regressions were found in some senders.
This release addresses all of those. We are close to there being none left.

There are also improvements in this release:

• #1350 Add “down” and “disconnected” run states to sr3 status
• #1323 DESTFN=NONE::TIME now works.
• #1299 after_gather entry point added.
• #1289 accept statements now provide arguments to plugins
• change in post_exchangeSplit… now based on relPath, not checksum. (work on HPC.)
• fix #1241 sr3 status more compact.
• fix #1206 exclude retries from lag calculation.
• #1301 sr3 status display fractional message rates when (< 1 / second.)
• user mode systemd unit file now tested and working.
• The https://dd.weather.gc.ca web site was revised. Example subscriptions revised to match.
• #1346 warn that queue parameters should be earlier in configs now.
• #1358 date/time duration breakage by @petersilva in #1359

Regressions compared to v2 that are fixed:

• #1337 mirror per umask, not global (sender)
• #1340 rename not working. (sender)
• #1327 inflight NONE in sender causes messages to be retried. (sender)
• #1322 sr3 sender fails to post message when no identity field in message.
• #1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362

HPC mirroring related fixes:

• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• #1297 part2 ordering mdelaylatest by @petersilva in #1368

What's Changed

• 3.00.54 maint by @petersilva in #1179
• got github notice about artifact@v3 being obsoleted by @petersilva in #1287
• Fix issue #1241 - make sr3 status display more compact by @mshak2 in #1290
• Add fileSizeMax to sender #1285 by @andreleblanc11 in #1286
• Check if there is anything to retry before figuring out if we by @petersilva in #1292
• systemd integration hints were v2 oriented by @petersilva in #1293
• Add method to recycle unused instance numbers #1288 by @andreleblanc11 in #1294
• Issue339 try5 fail connections in main loop by @petersilva in #1295
• issue #624 adding INSTANCE to variable substitution by @petersilva in #1296
• Issue634 design documentation review by @petersilva in #1303
• fix #1206 change send signature to support permanent failures by @petersilva in #1305
• Implement #1289 by @reidsunderland in #1306
• Fix #1307 - Decrease info-message length for sr3 cleanup by @mshak2 in #1313
• Fix #1301 by @mshak2 in #1312
• Fix #1309 by @andreleblanc11 in #1310
• exclude retries from lag accumulation by @petersilva in #1304
• Issue1299 - New after_gather entry point + better error handling in sundew_dirPattern by @andreleblanc11 in #1308
• user mode systemctl start should not setuid/setgid by @petersilva in #1317
• get DESTFN=NONE::TIME to work... I think. by @petersilva in #1323
• Fix #1321, allow any component's default_options overrides to work by @reidsunderland in #1326
• Issue1322 post exchange split revamp by @petersilva in #1329
• sender options inflight test fails if None by @petersilva in #1330
• Fix py3.6 crash by re-implementing #1289 by @reidsunderland in #1332
• fix partitions, broken because naturalsize output has no spaces anymore by @petersilva in #1333
• Fix #1325 by handling the exception by @reidsunderland in #1335
• Refactor email plugin config syntax by @reidsunderland in #1338
• Four in 1: Issue #1300 watch file perm, #1316 crash in download decision, #1254 poll fileAgeMin, #1337 mirror is not global by @petersilva in #1336
• fix #1283 disable new config by @petersilva in #1343
• user service cannot depend on multi-user (system) target by @petersilva in #1339
• Many small issues by @petersilva in #1342
• Fixing unit tests 202412 by @petersilva in #1345
• Issue1346 deal with change in semantics for queue declares from previous versions. by @petersilva in #1349
• changing topics for new citypage scheme by @petersilva in #1341
• Fix DiskQueue bugs by @reidsunderland in #1356
• sr3 status fixes #1350 and #1352. by @petersilva in #1355
• Issue1358 date/time duration breakage by @petersilva in #1359
• Add logic from gitlab version of plugins by @andreleblanc11 in #1361
• Issue1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362
• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• Issue1297 part2 ordering mdelaylatest by @petersilva in #1368

Full Changelog: v3.00.56...v3.00.57

Packages Available

(coming soon) Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.57/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• Experimental self-extracting windows installer also attached. WSL is preferred if possible.

v3.0.57rc2

What's Changed

• #1358 date/time duration breakage by @petersilva in #1359
• Add logic from gitlab version of plugins by @andreleblanc11 in #1361
• #1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362
• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• #1297 part2 ordering mdelaylatest by @petersilva in #1368

Full Changelog: v3.00.57rc1...v3.0.57rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.57rc2/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.57rc1 - closer to fine

Summary

"The less I seek my source for some definitive, the closer I am to fine." -- Indigo Girls.

The last version (3.0.56) was the first one where we thought sr3 could replace any
v2 configuration. As we did so, a few new regressions were found in some senders.
This release addresses all of those. We are close to there being none left.

There are also improvements in this release:

• #1350 Add “down” and “disconnected” run states to sr3 status
• #1323 DESTFN=NONE::TIME now works.
• #1299 after_gather entry point added.
• #1289 accept statements now provide arguments to plugins
• change in post_exchangeSplit… now based on relPath, not checksum. (work on HPC.)
• fix #1241 sr3 status more compact.
• fix #1206 exclude retries from lag calculation.
• #1301 sr3 status display fractional message rates when (< 1 / second.)
• user mode systemd unit file now tested and working.
• The https://dd.weather.gc.ca web site was revised. Example subscriptions revised to match.
• #1346 warn that queue parameters should be earlier in configs now.

Sender regressions compared to v2 that are fixed:

• #1337 mirror per umask, not global
• #1340 rename not working.
• #1327 inflight NONE in sender causes messages to be retried.
• #1322 sr3 sender fails to post message when no identity field in message.

As well as many other bug fixes.

Full Changelog: v3.00.56...v3.00.57rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.57rc1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56 Recursion is Back Baby, Subtopics After Queues

What's new

• CHANGE: queueName and/or queueShare must be above subtopic in configuration files. (order did not matter in earlier versions)
• CHANGE: in API breakage: replace sarracenia.credentials with sarracenia.config.credentials
• New: standby mode, triggered by attempts 0 in the configuration file, allows clean local queueing of transfers during long outages.
• New: recursion option for polls and watches (actually it was removed before, but added back.)
• New: logDuplicates option to show only files rejected by the duplicate suppression mechanisms.
• rest is fixes of many things. It just behaves better.
  • http download performance regression fixed.
  • ack failure recovery regression fixed.
  • HPC mirroring rename/symlink fix (requires metpx-sr3c 3.24.11 )

What's Changed

• Issue1199 - directory / not interpreted properly. by @petersilva in #1231
• just reducing bogus messages by @petersilva in #1230
• Unquote secret_access_key in s3CloudSender plugin by @reidsunderland in #1228
• sr3 sanity behaves badly with instances > 100. by @petersilva in #1226
• ANOTHER AM U18.04 adaptation + another AM sender problem found in satnet testings by @andreleblanc11 in #1221
• fix crash on sr3c pidfiles (i01.pid) by @reidsunderland in #1234
• Issue35 try 2 stateful binding part1 by @petersilva in #1232
• Switch from global QoS to per-consumer QoS by @reidsunderland in #1235
• Doc sec zoning multihop by @petersilva in #1236
• Fix bug from moving credentials into config subdir by @reidsunderland in #1239
• fix #1242 sftp making subdirs when it shouldn't by @reidsunderland in #1243
• Fix for #1156 by @mshak2 in #1193
• QoS should apply to all instances by @reidsunderland in #1245
• add ncftp client to basic vm config... might fix some flow tests. by @petersilva in #1249
• Json logging (EXPERIMENTAL) by @petersilva in #1238
• Add recursion back by @petersilva in #1237
• fix #1246 delete source when no options given by @petersilva in #1250
• Fixing unit tests by @petersilva in #1251
• testing: add pyftpdlib to apt packages to try avoiding version incompatibility by @reidsunderland in #1252
• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276
• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.55...v3.00.56

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.56/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56rc3 back to recursion, subtopics after queues

What's Changed

• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.56rc2...v3.00.56rc3

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc3/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.56rc2 back to recursion, subtopics after queues

Regression

• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.54.post1 )
  (stable version will not be released without a fix.)

What's New

• need to put queueName settings before the subtopic settings in configuration files, for this and all future versions.
• fixed #1261 performance regression for https download
• HPC rename/symlink mirroring fix: needs to be combined with an upcoming C (metpx-sr3c) package (relative symbolic links and renames when done from a directory that is symlinked to the tree being mirrored.)

What's Changed

• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276

Full Changelog: v3.00.56rc1...v3.00.56rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc2/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56rc1 back to recursion

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.54post1 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.54.post1 )
  (stable version will not be released without fix.)

What's New

• New: recursion option for polls and watches (actually it was removed before, but added back.)
• New: logDuplicates option to show only files rejected by the duplicate suppression mechanisms.
• rest is fixes of many things. It just behaves better.

What's Changed

• Issue1199 - directory / not interpreted properly. by @petersilva in #1231
• just reducing bogus messages by @petersilva in #1230
• Unquote secret_access_key in s3CloudSender plugin by @reidsunderland in #1228
• sr3 sanity behaves badly with instances > 100. by @petersilva in #1226
• ANOTHER AM U18.04 adaptation + another AM sender problem found in satnet testings by @andreleblanc11 in #1221
• fix crash on sr3c pidfiles (i01.pid) by @reidsunderland in #1234
• Issue35 try 2 stateful binding part1 by @petersilva in #1232
• Switch from global QoS to per-consumer QoS by @reidsunderland in #1235
• Doc sec zoning multihop by @petersilva in #1236
• Fix bug from moving credentials into config subdir by @reidsunderland in #1239
• fix #1242 sftp making subdirs when it shouldn't by @reidsunderland in #1243
• Fix for #1156 by @mshak2 in #1193
• QoS should apply to all instances by @reidsunderland in #1245
• add ncftp client to basic vm config... might fix some flow tests. by @petersilva in #1249
• Json logging (EXPERIMENTAL) by @petersilva in #1238
• Add recursion back by @petersilva in #1237
• fix #1246 delete source when no options given by @petersilva in #1250
• Fixing unit tests by @petersilva in #1251
• testing: add pyftpdlib to apt packages to try avoiding version incompatibility by @reidsunderland in #1252

Full Changelog: v3.00.55...v3.00.56rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.55post1

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.56 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.56 )

What's Changed

• fix #1242 regression.

Full Changelog: v3.00.55...o3.00.55post1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.55.post1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.55 Good Timing

REGRESSION

• known performance regression on http downloads #1261 (work-around: use v3.00.56 )
• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.56 )

Briefly:

Working through deployment issues on core operational data pumps:

• fixing more timers expiring during transfers. many reliability improvements.
• many more polls deployed, and some improvements for them.
• Add scheduled_time option for scheduled flows to run at the given time.
• management UI improvements (sr3 status display of lag, queue sr3 stop works better.)
• fix for citypage download issues (and other datasets that rapidly change upstream.)
• scheduled flow is simplified and more accurate.
• fix #1219 time offsets in configuration files now correctly converted

What's Changed

• Add new BC Trans API to sr3 by @andreleblanc11 in #1150
• Fix #1151 by @andreleblanc11 in #1152
• Fix #953 re-enable and fix passive queue declare by @reidsunderland in #1160
• AM adaptation for U18.04 servers by @andreleblanc11 in #1164
• The Hearty eXpert Issue1138 - documenting messageAgeMax by @petersilva in #1147
• Issue1159 by @petersilva in #1162
• Later pip needs convincing by @petersilva in #1171
• Issue1155 AMQP ack refactor by @reidsunderland in #1174
• #1167 fix clash in class names with a directory prefix. inelegant, bu… by @petersilva in #1172
• Fix #1173: problems stopping instances by @reidsunderland in #1178
• fix #1166 improved download decision criteria by @petersilva in #1175
• fix Ubuntu 20.04 flow tests by @reidsunderland in #1186
• Implement HTTP(S) stat using urllib, and HTTPS transfer class minor refactor by @reidsunderland in #1202
• Change stop output to show the number of processes still running (like v2) by @reidsunderland in #1204
• fix rejectPercent maybe other stuff? by @petersilva in #1192
• maybe fix #1195 more log fields by default by @petersilva in #1197
• Fix badly expiring timers by @petersilva in #1208
• Add scheduled_time for #1209 by @andreleblanc11 in #1210
• Issue1198 - charset option for ftp. by @petersilva in #1200
• Add more wiggle room in scheduled_flow for accumulated wait overhead by @andreleblanc11 in #1217
• Fix for Issue1218 - Permission problems in directory creations on FTP/SFTP transfers by @andreleblanc11 in #1220
• Issue1214 another scheduled refactor by @petersilva in #1223
• fix #1219 enough to convert ops by @petersilva in #1224

Full Changelog: v3.00.54...v3.00.55

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.55/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.55rc2

What's Changed

• Issue1214 another scheduled refactor by @reidsunderland in #1223
• fix #1219 enough to convert ops by @petersilva in #1224

Full Changelog: v3.00.55rc1...v3.00.55rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.55rc2/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.