8.0.8 (21 July 2023)

Announcements

Small change to further reduce memory usage by not creating unused data
structures.

This revealed a glitch with dry-runs that would eventually have been reported.

Changes

• Don't populate IndexList.index_info until required by a filter. In other
  words, stop populating the zero values as part of instantiation.
• This uncovered an oversight with the 8.0.7 release. Certain actions, if
  taken with no filters, or only a pattern filter, would never ever populate
  the index_info. This wasn't a huge problem, unless you were using the
  dry-run flag. The dry-run output requires the index_info to be present for
  each index. In 8.0.7, where the empty structure was already in place, a
  dry-run wouldn't fail, but it also wouldn't show any data. This is fixed.
• A few tests also needed updating. They were using contrived scenarios to
  test certain conditions. Now these tests are manually grabbing necessary
  metadata so they can pass.

8.0.7 (21 July 2023)

Announcements

There are no changes in this release that change how Curator will appear to behave to the user. However...

This release ends the practice of collecting all stats and metadata at IndexList initiation. This should make execution much faster for users with enormous clusters with hundreds to thousands of indices. In the past, this was handled at IndexList instantiation by making a cluster state API call. This is rather heavy, and can be so much data as to slow down Curator for minutes on clusters with hundreds to thousands of shards. This is all changed in this release.

For example, the pattern filter requires no index metadata, as it only works against the index name. If you use a pattern filter first, the actionable list of indices is reduced. Then if you need to filter based on age using the creation_date, the age filter will call get_index_settings to pull the necessary data for that filter to complete. Some filters will not work against closed indices. Those filters will automatically call get_index_state to get the open/close status of the indices in the actionable list. The disk space filter will require the index state as it won't work on closed indices, and will call get_index_stats to pull the size_in_bytes stats.

Additionally, the cat API is used to get index state (open/close), now, as it is the only API call besides the cluster state which can inform on this matter. Not calling for a huge dump of the entire cluster state should drastically reduce memory requirements, though that may vary for some users still after all of the index data is polled, depending on what filters are used.

There is a potential caveat to all this rejoicing, however. Searchable snapshot behavior with ILM policies usually keeps indices out of Curator's vision. You need to manually tell Curator to allow it to work on ILM enabled indices. But for some users who need to restore a snapshot to remove PII or other data from an index, it can't be in ILM anymore. This has caused some headaches. For example, if you are tracking an index in the hot tier named 'index1' and it is in process of being migrated to the cold tier as a searchable snapshot, it may suddenly disappear from the system as 'index1' and suddenly re-appear as 'restored-index1'. The original index may now be an alias that points to the newly mounted cold-tier index. Before this version, Curator would choke if it encountered this scenario. In fact, one user saw it repeatedly. See the last comment of issue 1682 in the GitHub repository for more information.

To combat this, many repeated checks for index integrity have become necessary. This also involves verifying that indices in the IndexList are not actually aliases. Elasticsearch provides exists tests for these, but they cannot be performed in bulk. They are, however, very lightweight. But network turnaround times could make large clusters slower. For this reason, it is highly recommended that regex filters be used first, early, and often, before using any other filters. This will reduce the number of indices Curator has to check and/or verify during execution, which will speed things up drastically.

Other Changes

Most of the unit tests needed updating due to these changes (adding a few more Mock responses), so I took the time to refactor them to use a builder method instead of repeating the same few lines of code in each test. This represents the vast majority of code deletions (2,085 lines!). All of the tests are passing, both integration and unit tests.

8.0.6 (18 July 2023)

Breakfix Release

• Small breakfix change to catch a similar rare race condition patched in
  8.0.5 covering the get_index_stats() method of IndexList. This patch
  covers the get_metadata() method and closes #1682.

8.0.5 (13 July 2023)

Announcements

Release for Elasticsearch 8.8.2

Changes

• Small PEP formatting changes that were found editing code.
• Bump Python version in Dockerfile to 3.11.4
• Bump Python dependency versions.
• Change targetName to target_name in setup.py for newest version
  of cx_Freeze. Hat tip to @rene-dekker in #1681 who made these changes
  to 5.x and 7.x.
• Fix command-line behavior to not fail if the default config file is not
  present. The newer CLI-based configuration should allow for no config file
  at all, and now that's fixed.
• Initial work done to prevent a race condition where an index is present at IndexList
  initialization, but is missing by the time index stats collection begins. The resultant
  404s were causing Curator to shut down and not complete steps.
• When running in a Docker container, make Curator log to /proc/1/fd/1 by
  default, if no value is provided for logfile (otherwise, use that).

8.0.4 (28 April 2023)

Announcements

Allow single-string, base64 API Key tokens in Curator.

To use a base64 API Key token in YAML configuration:

  elasticsearch:
    client:
      hosts: https://host.example.tld:9243
    other_args:
      api_key:
        token: '<base64 token goes here>'

To use a base64 API Key token at the command-line:

  curator --hosts https://host.example.tld:9243 --api_token <base64 token goes here> [OTHER ARGS/OPTIONS]

NOTE: In neither of the above examples are the alligator clips necessary (the < and > characters).

Changes

• Update es_client to 8.7.0, which enables the use of the base64 encoded API Key token.
  This also fixes #1671 via untergeek/es_client#33

8.0.3 (22 February 2023)

Announcements

A new action called cold2frozen has been added to Curator. It is not going to be of much use to
the vast majority of Elasticsearch users as it serves a very narrow use-case. That is, it migrates
searchable snapshot indices from the cold tier to the frozen tier, but only if they are not
associated with ILM (Index Lifecycle Management) policies. As escalation into the cold and frozen
tiers is usually handled by ILM, this is indeed a rare use case.

Changes

• Fixed instruction display for delete repository action of es_repo_mgr
• Fix unit tests to import more specifically/cleanly
• Fixed Hatch build includes (this was speed-released to PyPI as 8.0.2.post1) as Curator did not
  function after a pip install.
• Added cold2frozen action, and tests.

8.0.2 (15 February 2023)

Changes

• Added the same CLI flags that the singletons offers. This gives much more flexibility with
  regards to passing configuration settings as command-line options, particularly for Docker.
• Re-created the get_client function. It now resides in curator.helpers.getters and will
  eventually see use in the Reindex class for remote connections.
• Created a new set of classes to import, validate the schema, and split individual actions into
  their own sub-object instances. This is primarily to make curator/cli.py read much more
  cleanly. No new functionality here, but fewer conditional branches, and hopefully more readable
  code.
• Updated the documentation to show these changes, both the API and the Elastic.co usage docs.

8.0.1 (10 February 2023)

Announcements

The 8.0.0 release was about getting Curator out the door with all of the functionality users were
accustomed to in 5.8.4, but with the newer, updated args and methods in elasticsearch8. Very
little else was changed that didn't need to be. Now comes a few improvements, and more are coming,
which is why I didn't start with 8.6.0 as my release version.

• Now offering multi-architecture Docker builds for arm64 (v8) and amd64.
• This required the addition of two new scripts at the root level of the project:
  alpine4docker.sh and post4docker.py. These scripts are used only when building the
  Dockerfile. They were needed to make multi-architecture Docker images possible. I'm sure you'll
  be able to see how they work with a cursory glance.

Breaking Changes

• I split curator.utils into several, separate modules under curator.helpers.

  I suppose, technically, that this qualifies as a breaking change from 8.0, but I sincerely
  doubt I have any users using Curator as an API yet, so I made the change. No functions were
  renamed, so this isn't as breaking so much as a slight shift in module naming. This gave me
  headaches, but it needed to be done a long time ago. It was always grating to see the Pylint
  warnings that the file is longer than 1000 lines, and searching for the module you wanted was
  way too much scrolling. This also gave me the chance to update the tests and the docstring's
  formatting for rST docs. Most of this release's changes came from this change.

Changes

• Curator has supported ECS logging for a while, but now that there is an official Python module,
  Curator is going to use it. Welcome, ecs-logging! As before, just use logformat: ecs,
  but now it has all of the goodness right there!
• rST docs are improved and updated. Check out https://curator.readthedocs.io to see.
• Logging turned out to be too verbose due to a shift. Now the blacklist defaults to
  ['elastic_transport', 'urllib3']. Documentation updated accordingly.
• Default behavior is now to not verify snapshot repository access for Snapshot and Restore
  actions. It was a hacky fix for older versions of Elasticsearch that just shouldn't be needed.

8.0.0 (31 January 2023)

Announcement

This release is a major refactoring of the Curator code to work with both Elasticsearch
8.x and the Elasticsearch-py Python module of the same major and minor versions.

I apologize for the crazy merge messes trying to get this all to work. In the end, I had to delete
my fork on github and start over clean.

Breaking Changes

• Curator is now version locked. Curator v8.x will only work with Elasticsearch v8.x
• Your old Curator config.yml file will no longer work as written. There have been more than
  a few changes necessitated by the updates in the elasticsearch8 Python client library.
  The client connection code has also been extracted to its own module, es_client. This is
  actually a good thing, however, as new options for configuring the client connection become
  possible.
• Going forward, Curator will only be released as a tarball via GitHub, as an sdist or
  wheel via pip on PyPI, and to Docker Hub. There will no longer be RPM, DEB, or Windows
  ZIP releases. I am sorry if this is inconvenient, but one of the reasons the development and
  release cycle was delayed so long is because of how painfully difficult it was to do releases.
• Curator will only work with Python 3.8+, and will more tightly follow the Python version releases.

Changes

• Last minute doc fixes. Mostly updated links to Elasticsearch documentation.
• Python 3.11.1 is fully supported, and all versions of Python 3.8+ should be fully supported.
• Use hatch and hatchling for package building & publishing
• Because of hatch and pyproject.toml, the release version still only needs to be tracked
  in curator/_version.py.
• Maintain the barest setup.py for building a binary version of Curator for Docker using
  cx_Freeze.
• Remove setup.cfg, requirements.txt, MANIFEST.in, and other files as functionality
  is now handled by pyproject.toml and doing pip install . to grab dependencies and
  install them. YAY! Only one place to track dependencies now!!!
• Preliminarily updated the docs.
• Migrate towards pytest and away from nose tests.
• Revamped almost every integration test
• Scripts provided now that aid in producing and destroying Docker containers for testing. See
  docker_test/scripts/create.sh. To spin up a numbered version release of Elasticsearch, run
  docker_test/scripts/create.sh 8.6.1. It will download any necessary images, launch them,
  and tell you when it's ready, as well as provide REMOTE_ES_SERVER environment variables for
  testing the reindex action, e.g.
  REMOTE_ES_SERVER="http://172.16.0.1:9201" pytest --cov=curator. These tests are skipped
  if this value is not provided. To clean up afterwards, run docker_test/scripts/destroy.sh
• The action classes were broken into their own path, curator/actions/filename.py.
• curator_cli has been updated with more client connection settings, like cloud_id.
• As Curator 8 is version locked and will not use AWS credentials to connect to any ES 8.x
  instance, all AWS ES connection settings and references have been removed.

8.0.0rc1 (30 January 2023)

Announcement

This release-candidate is a major refactoring of the Curator code to work with both Elasticsearch
8.x and the Elasticsearch-py Python module of the same major and minor versions.

Breaking Changes

• Curator is now version locked. Curator v8.x will only work with Elasticsearch v8.x
• Your old Curator config.yml file will no longer work as written. There have been more than
  a few changes necessitated by the updates in the elasticsearch8 Python client library.
  The client connection code has also been extracted to its own module, es_client. This is
  actually a good thing, however, as new options for configuring the client connection become
  possible.
• Going forward, Curator will only be released as a tarball via GitHub, as an sdist or
  wheel via pip on PyPI, and to Docker Hub. There will no longer be RPM, DEB, or Windows
  ZIP releases. I am sorry if this is inconvenient, but one of the reasons the development and
  release cycle was delayed so long is because of how painfully difficult it was to do releases.
• Curator will only work with Python 3.8+, and will more tightly follow the Python version releases.

Changes

• Python 3.11.1 is fully supported, and all versions of Python 3.8+ should be fully supported.
• Use hatch and hatchling for package building & publishing
• Because of hatch and pyproject.toml, the release version still only needs to be tracked
  in curator/_version.py.
• Maintain the barest setup.py for building a binary version of Curator for Docker using
  cx_Freeze.
• Remove setup.cfg, requirements.txt, MANIFEST.in, and other files as functionality
  is now handled by pyproject.toml and doing pip install . to grab dependencies and
  install them. YAY! Only one place to track dependencies now!!!
• Preliminarily updated the docs.
• Migrate towards pytest and away from nose tests.
• Revamped almost every integration test
• Scripts provided now that aid in producing and destroying Docker containers for testing. See
  docker_test/scripts/create.sh. To spin up a numbered version release of Elasticsearch, run
  docker_test/scripts/create.sh 8.6.1. It will download any necessary images, launch them,
  and tell you when it's ready, as well as provide REMOTE_ES_SERVER environment variables for
  testing the reindex action, e.g.
  REMOTE_ES_SERVER="http://172.16.0.1:9201" pytest --cov=curator. These tests are skipped
  if this value is not provided. To clean up afterwards, run docker_test/scripts/destroy.sh
• The action classes were broken into their own path, curator/actions/filename.py.
• curator_cli has been updated with more client connection settings, like cloud_id.
• As Curator 8 is version locked and will not use AWS credentials to connect to any ES 8.x
  instance, all AWS ES connection settings and references have been removed.