v3.00.53pre2 - Cloud Transfers Enabled

• Introduces (experimental) S3 and Azure Transfer classes (native support for polling and transfer to and from cloud storage.) from (@gcglinton !). but also many other improvements.
• heavily re-factored and improved AM server.
• heavily re-factored poll
• improvements in callback api.

What's Changed

• Add topicPrefix to docs to fix #961 by @reidsunderland in #964
• Fixed "Downloading using CLI" tutorial page by @mshak2 in #962
• Remove threading from AMQP Consumer code by @reidsunderland in #965
• Run housekeeping during long sleeps by @reidsunderland in #969
• fix #967 define hostname as a string option, and stuff works by @petersilva in #970
• Add missing tokIsYear method from Sundew. Prevents some erronous bulletins not being filtered. by @andreleblanc11 in #973
• fix #971 Add missing to list of possible process status by @petersilva in #972
• Re-enable AMQP consumer flow tests by @reidsunderland in #977
• Reject float types for Julian day value in getTime by @andreleblanc11 in #979
• Housekeeping logging changes by @reidsunderland in #978
• Rename tests to have consistent naming by @reidsunderland in #981
• Add S3 Sender plugin by @reidsunderland in #983
• NASA PODAAC checksum change by @reidsunderland in #988
• Issue986 - stop converting directory to path in poll. by @petersilva in #987
• Issue992 - re-parse values using correct type for add_options from flow_callbacks. by @petersilva in #999
• No download message when not downloading by @petersilva in #998
• fix #982 by checking for statehost in cleanup by @petersilva in #996
• Issue994 unit tests broken and failing alot (but weirdly not always.) by @petersilva in #995
• Clarify a bit more how each component works in the documentation by @andreleblanc11 in #993
• Issue997 - Eliminate retry file race condition and make AM work more like a sarracenia flow by @andreleblanc11 in #1005
• Issue974 yet another scheduled_poll refactor PR by @petersilva in #991
• Transfer documentation and a fix for all tests... by @petersilva in #1009
• #982 statehost config option should be used to see pick state directory by @petersilva in #1003
• Native S3 transfer driver by @gcglinton in #1010
• Make persistent a configurable option (sr3) by @reidsunderland in #1013
• Azure transfer driver by @gcglinton in #1019
• EUMETSAT poll plugin update by @reidsunderland in #1027
• implement topicCopy for #1023 by @petersilva in #1025
• fix #947, only report on .conf or .inc files by @petersilva in #1014
• Issue1022 by @petersilva in #1024
• Wis2 integration - MQTT publishing using sr3 stack by @mshak2 in #1026
• Issue1029 wmo00 accumulated file format support by @petersilva in #1030
• Issue908 - Improve general am/gather code and properly kill children zombies. by @andreleblanc11 in #1031
• flakey flow test failing when polls killed with -9 cannot remove novip state file by @petersilva in #1032
• Issue980 - Add verifyHeader method to prevent erronous headers from passing through by @andreleblanc11 in #1033
• updatePaths work without being supplied new_dir and new_path by @petersilva in #1034
• locally installed plugins are not selected because we prepend system … by @petersilva in #1038
• Exchange split override by @petersilva in #1041
• updatePaths no longer fails when called with just options by @petersilva in #1039
• Better sender io errors (download also...) by @petersilva in #1037

Full Changelog: v3.00.52...v3.00.53pre2

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.53rc2/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Enterprise Linux Version 8.9 (rhel8_9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.53rc1 - S3 driver, AM server, Poll enhancements

• New (experimental) S3 Transfer Protocol driver.
• heavily re-factored and improved AM server.
• heavily re-factored poll

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

What's Changed

• Add topicPrefix to docs to fix #961 by @reidsunderland in #964
• Fixed "Downloading using CLI" tutorial page by @mshak2 in #962
• Remove threading from AMQP Consumer code by @reidsunderland in #965
• Run housekeeping during long sleeps by @reidsunderland in #969
• fix #967 define hostname as a string option, and stuff works by @petersilva in #970
• Add missing tokIsYear method from Sundew. Prevents some erronous bulletins not being filtered. by @andreleblanc11 in #973
• fix #971 Add missing to list of possible process status by @petersilva in #972
• Re-enable AMQP consumer flow tests by @reidsunderland in #977
• Reject float types for Julian day value in getTime by @andreleblanc11 in #979
• Housekeeping logging changes by @reidsunderland in #978
• Rename tests to have consistent naming by @reidsunderland in #981
• Add S3 Sender plugin by @reidsunderland in #983
• NASA PODAAC checksum change by @reidsunderland in #988
• Issue986 - stop converting directory to path in poll. by @petersilva in #987
• Issue992 - re-parse values using correct type for add_options from flow_callbacks. by @petersilva in #999
• No download message when not downloading by @petersilva in #998
• fix #982 by checking for statehost in cleanup by @petersilva in #996
• Issue994 unit tests broken and failing alot (but weirdly not always.) by @petersilva in #995
• Clarify a bit more how each component works in the documentation by @andreleblanc11 in #993
• Issue997 - Eliminate retry file race condition and make AM work more like a sarracenia flow by @andreleblanc11 in #1005
• Issue974 yet another scheduled_poll refactor PR by @petersilva in #991
• Transfer documentation and a fix for all tests... by @petersilva in #1009
• #982 statehost config option should be used to see pick state directory by @petersilva in #1003
• Native S3 transfer driver by @gcglinton in #1010
• Make persistent a configurable option (sr3) by @reidsunderland in #1013

Full Changelog: v3.00.52...v3.00.53rc1

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.53rc1/
Note: uploaded pypi package from ubuntu 22.04... installation on older OS & python versions may
be compromised (error, will be corrected in next release.)

• (Coming Soon) RPM package for fedora core 39 (fc39)

• (Coming Soon) RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.

v2.24.04 prevent message loss on broker restart

NOTE: version 2 stream is the legacy support version of Sarracenia. For any new work, please look at metpx-sr3 instead.

Release Notes

Primary rationale for this release is to make it easier to restart nodes in clusters. We noticed there is message loss on broker restart. Switching to persistent messaging by default with options to re-instate old behaviour. Likely to validate performance
by turning options on and off and comparing.

• from PR #1012 enable publishing of persistent messages (withstand broker restart), combination of: durable and persistent
• fixed msg_fdelay plugin to work as documented.
• part of #639 take root of paths from baseUrl (formerly ignored) compatibility with sr3 (not used in v2 until now.)
• part of #639 ignore mkdir and rmdir events without errors (unimplemented in v2)

Noticed that v2.22.10 was the last release publicized on github. There were interim releases on launchpad, which were
never publicized on github. Including those for completeness.

following changes from v2.22.10p1, v2.23.06, v2.23.07:

• fixes for #715 missing attribute failure to process message
• fix #703 replacing "integrity" with "identity" in v03 format (compatibility with sr3)
• from PR #655 graceful degradation when dateparser module missing.
• fix #645 dependency for systemd startup on network-online
• fix #382 sr cleanup to only remove unused exchanges.
• dd_subscribe removed (wasn't working, not really supported.)
• minor fixes.
• v2 fallback code so it partially works even in watchdog module is missing.

BUG: the version tag was not properly updated, so in pypi, this version is v2.23.12 If installing from launchpad,
the package will be externally named 2.24.04, but post installation, it will also claim to be v2.23.12.

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

Publishing only to launchpad pre-release repository for (tentatively) one week. Once the pre-release phase
is done, will upload to stable repo, and pypi.

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-release
python: https://pypi.org/project/metpx-sarracenia/

v3.00.52

What's Changed

• correct work description for different running components by @petersilva in #921
• fix #922 avoid mixing up post_exchange with exchange in display by @petersilva in #923
• changing defaults and documenting the consideration by @petersilva in #924
• fix #907 admin declare exchange by @petersilva in #930
• Update subscriber.rst by @petersilva in #929
• Issue920 fix inflight mtime not working. by @petersilva in #926
• Issue912 retry refilter flag by @petersilva in #928
• Issue920 cleanup and #912 add _isRetry by @petersilva in #932
• make rename field processing contingent on download option being set by @petersilva in #936
• Issue918 - Add plugin to rename AM bulletins properly by @andreleblanc11 in #933
• adding mention of scheduled flows as an option when migrating v2 polls by @petersilva in #938
• fix #940 brackets wrong by @petersilva in #941
• Issue942 - sarra consumer writing wrong files because 'rename' field should have been deleted. by @petersilva in #944
• Issue934 2 reducing references to v2 (sr_subscribe, sr_watch, etc...) in documentation. by @petersilva in #937
• Lots of AM corrections (Renamer + AHL corrections in gather) by @andreleblanc11 in #946
• Am final touches before first release by @andreleblanc11 in #949
• Edit for typos and clarity by @mshak2 in #950
• Move AM renamer to gather. Allows us to properly use sundew variable expansions. by @andreleblanc11 in #952
• Issue953 PRECONDITION_FAILED unknown delivery tag by @petersilva in #954
• Properly handle trailing spaces at end of bulletin headers by @andreleblanc11 in #956
• Change raise_signal to os.kill for #948 by @reidsunderland in #957
• Add msg['post_baseUrl'] processing for #951 by @reidsunderland in #958
• Add plugin to poll the Copernicus Marine Data Store service by @reidsunderland in #959
• fix #955 check cache dir exists before cleaning it. by @petersilva in #960
• Add fix to handle improperly parsed station names from getStation by @andreleblanc11 in #963

Full Changelog: v3.00.51...v3.00.52

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.52/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.

v3.00.52rc4

What's Changed

• Change raise_signal to os.kill for #948 by @reidsunderland in #957
• Add msg['post_baseUrl'] processing for #951 by @reidsunderland in #958
• Add plugin to poll the Copernicus Marine Data Store service by @reidsunderland in #959
• fix #955 check cache dir exists before cleaning it. by @petersilva in #960
• Add fix to handle unproperly parsed station names from getStation by @andreleblanc11 in #963

Full Changelog: v3.00.52rc3...v3.00.52rc4

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc4/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.52rc3

What's Changed

• Edit for typos and clarity by @mshak2 in #950
• Move AM renamer to gather. Allows us to properly use sundew variable expansions. by @andreleblanc11 in #952
• Issue953 PRECONDITION_FAILED unknown delivery tag by @petersilva in #954
• Properly handle trailing spaces at end of bulletin headers by @andreleblanc11 in #956

New Contributors

• @mshak2 made their first contribution in #950

Full Changelog: v3.00.52rc2...v3.00.52rc3

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc3/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.52rc2

What's Changed

• adding mention of scheduled flows as an option when migrating v2 polls by @petersilva in #938
• fix #940 brackets wrong by @petersilva in #941
• Issue942 - sarra consumer writing wrong files because 'rename' field should have been deleted. by @petersilva in #944
• Issue934 2 reducing references to v2 (sr_subscribe, sr_watch, etc...) in documentation. by @petersilva in #937
• Lots of AM corrections (Renamer + AHL corrections in gather) by @andreleblanc11 in #946
• Am final touches before first release by @andreleblanc11 in #949

Full Changelog: v3.0.52rc1...v3.00.52rc2

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc2/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.0.52rc1 watch working with inflight times.

What's Changed

• correct work description for different running components by @petersilva in #921
• fix #922 avoid mixing up post_exchange with exchange in display by @petersilva in #923
• changing defaults and documenting the consideration by @petersilva in #924
• fix #907 admin declare exchange by @petersilva in #930
• Update subscriber.rst by @petersilva in #929
• Issue920 fix inflight mtime not working. by @petersilva in #926
• Issue912 retry refilter flag by @petersilva in #928
• Issue920 cleanup and #912 add _isRetry by @petersilva in #932
• make rename field processing contingent on download option being set by @petersilva in #936
• Issue918 - Add plugin to rename AM bulletins properly by @andreleblanc11 in #933

Full Changelog: v3.00.51rc7...v3.0.52rc1

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc1/

• RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

3.00.51

What's Changed

• V03 release process revision by @petersilva in #859
• fix #866 by @petersilva in #867
• turn off magic support if redhat library present. by @petersilva in #868
• Fix Issue 861 - variable substitution issues by @reidsunderland in #865
• Issue824 - last transfer (for files) last message (for message flows) in metrics and status. by @petersilva in #871
• Issue860 fixing sourceFromExchange by @petersilva in #870
• fix #872 crashing because o.identity is kind of a typo by @petersilva in #873
• Issue860 3 fix sourceFromExchange by @petersilva in #876
• Issue860 4 suppress the default source guess if some other option is specified. And more! by @petersilva in #877
• Add plugins and examples for retrieving data from Copernicus by @reidsunderland in #878
• Note about batch by @reidsunderland in #879
• close #880 implements list of vips, ORed. by @petersilva in #883
• closes #881 by removing files on download failure by @petersilva in #882
• reverting quoting in one place, to see if it helps by @petersilva in #885
• Issue740 3 Adding pyproject.toml support for better installation on python > 3.6 by @petersilva in #887
• Close gzip target file before renaming it (to ensure it's flushed) by @endlisnis in #889
• Issue884 second fix for #884, this time with safe= by @petersilva in #890
• New EUMETSAT plugins; updates for copernicus by @reidsunderland in #891
• Add plugin for rate limiting polls by @reidsunderland in #886
• fixing problem with inflight numeric value being ignored by @petersilva in #892
• Branch396 v3 51 re-enabling partitioned transfers on an experimental basis. by @petersilva in #888
• removing xattr as hard dependency (already registered as a feature) by @petersilva in #899
• promote do() entry point from subscribe to parent flow by @petersilva in #903
• Add AM AHL headers and put bulletin contents in message by @andreleblanc11 in #900
• Issue905 Queue Names Getting Clobberred. by @petersilva in #909
• Fix character encoding for AM and move code to gather by @andreleblanc11 in #916
• fix #910 add a further heuristic for detecting v02 messages by @petersilva in #911

Full Changelog: v3.00.50...3.00.51

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.51/

• RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.51rc7

What's Changed

• removing xattr as hard dependency (already registered as a feature) by @petersilva in #899
• promote do() entry point from subscribe to parent flow by @petersilva in #903
• Add AM AHL headers and put bulletin contents in message by @andreleblanc11 in #900
• fix #905 Queue Names Getting Clobberred. by @petersilva in #909
• Fix character encoding for AM and move code to gather by @andreleblanc11 in #916
• fix #910 add a further heuristic for detecting v02 messages by @petersilva in #911

Full Changelog: v3.00.51rc6...v3.00.51rc7

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.51rc4/

• attached: redhat package for fedora core 39 (fc39)

• attached: RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• attached: self-extracting windows installer also attached.