From 8b6c9b6020e8a0c8bfb0ea52c2fdca41e01415cd Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 21 May 2024 13:27:29 -0400 Subject: [PATCH 1/5] Adds information about tiering rules and exclusive access Partially addresses #1207 --- source/administration/object-management.rst | 27 +++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index a432167b..415f0d77 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -220,6 +220,33 @@ Conversion to or from XML may be required for importing rules created on S3 or s See :ref:`minio-lifecycle-management` for more complete documentation. +Target Bucket Configuration for Tiering +--------------------------------------- + +Keep in mind the following considerations when configuring the target bucket for tiering: + +- The target bucket *can* have its own set of object management rules different from the source. + For example, the source bucket may have object locking defined while the target bucket does not. +- While enabling tiering requires :ref:`versioning ` on the source bucket, the target bucket does *not* require versioning. + Enabling versioning on both the source and target buckets may lead to unexpected results. + +Exclusive Access to Remote Data +------------------------------- + +MinIO **must** have *exclusive* access to the target bucket. +No other user, process, application, or resource should have any access to or perform any actions against the target bucket. + +All access to the transitioned objects *must* occur through MinIO via S3 API operations only. +Manually modifying a transitioned object - whether the metadata on the “hot” MinIO tier or the object data on the remote “warm/cold” tier - may result in loss of that object data. + +MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions: + +- No external mutation, migration, or deletion of objects on the remote storage. +- No lifecycle management rules (such as transition or expiration) on the remote storage bucket. + +To facilitate this exclusive access, grant the lifecycle management user ``read``, ``write``, and ``delete`` access to the target bucket in its :ref:`policy `. +All other policies should ``deny`` access to the target bucket. + .. toctree:: :titlesonly: :hidden: From a47d748e2f21019b89f0a55a226d6b3e628e5fcf Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 21 May 2024 14:36:32 -0400 Subject: [PATCH 2/5] PR feedback --- source/administration/object-management.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index 415f0d77..a7b49722 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -220,13 +220,18 @@ Conversion to or from XML may be required for importing rules created on S3 or s See :ref:`minio-lifecycle-management` for more complete documentation. -Target Bucket Configuration for Tiering ---------------------------------------- +Target Bucket Considerations +---------------------------- Keep in mind the following considerations when configuring the target bucket for tiering: -- The target bucket *can* have its own set of object management rules different from the source. +- The target bucket *can* have its own set of object management rules. + Tiering does *not* require the source and target buckets have the same rules. + For example, the source bucket may have object locking defined while the target bucket does not. + + Target buckets should *not* have their own rules for expiration or additional tiering. + MinIO supports only one level of tiering. - While enabling tiering requires :ref:`versioning ` on the source bucket, the target bucket does *not* require versioning. Enabling versioning on both the source and target buckets may lead to unexpected results. From 756665157842296ef23ffa838d3446a91f1c15a5 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Tue, 21 May 2024 15:47:08 -0400 Subject: [PATCH 3/5] More PR feedback and rework --- source/administration/object-management.rst | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index a7b49722..3272257e 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -223,17 +223,22 @@ See :ref:`minio-lifecycle-management` for more complete documentation. Target Bucket Considerations ---------------------------- -Keep in mind the following considerations when configuring the target bucket for tiering: +- MinIO does *not* require that the target bucket match object management or versioning configurations with the source bucket. +- Object locking set on the target bucket may prevent desired ``delete`` operations from the source bucket from completing. +- If the target bucket has expiration rules configured, these may result in data loss. +- MinIO tiers objects with their own ``UUID``, so versioning on the target bucket is not required. + + Versioning on the target bucket may have the following unexpected or undesired results: + + - Excess data usage on the target remote after a delete operation on the source + - Reduced storage efficiency on the target, as ``delete`` operations result in ``DeleteMarker`` + - Duplicate delete markers on source and target buckets - The target bucket *can* have its own set of object management rules. - Tiering does *not* require the source and target buckets have the same rules. - For example, the source bucket may have object locking defined while the target bucket does not. - Target buckets should *not* have their own rules for expiration or additional tiering. MinIO supports only one level of tiering. -- While enabling tiering requires :ref:`versioning ` on the source bucket, the target bucket does *not* require versioning. - Enabling versioning on both the source and target buckets may lead to unexpected results. + Mismatched rules between the source and the target buckets run a risk of data corruption and loss. Exclusive Access to Remote Data ------------------------------- From e83a45715532fbc376613d313b775c9414653993 Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Wed, 22 May 2024 13:04:10 -0400 Subject: [PATCH 4/5] Reworking target bucket section --- source/administration/object-management.rst | 25 ++++++++++++--------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index 3272257e..60ac016e 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -223,23 +223,28 @@ See :ref:`minio-lifecycle-management` for more complete documentation. Target Bucket Considerations ---------------------------- -- MinIO does *not* require that the target bucket match object management or versioning configurations with the source bucket. +MinIO does *not* require that the target bucket match object management or versioning configurations with the source bucket. +The target bucket *may* have its own set of object management rules, if defined with care. + +Target buckets should *not* have their own rules for expiration or additional tiering. +Expiration rules can result in removal of tiered data still in use by the source bucket. +Tiering to an additional remote creates an additional network hop between the hot tier and it's data while also increasing operational complexity. + +You *may* configure object locking or versioning on the remote bucket. +Use caution, as mismatched rules between the source and the target buckets run a risk of data corruption and loss. + +Creating separate rules on the target bucket may have effects such as the following: + - Object locking set on the target bucket may prevent desired ``delete`` operations from the source bucket from completing. -- If the target bucket has expiration rules configured, these may result in data loss. + - MinIO tiers objects with their own ``UUID``, so versioning on the target bucket is not required. - Versioning on the target bucket may have the following unexpected or undesired results: + If you enable versioning on the target bucket, you may experience the following unexpected or undesired results: - Excess data usage on the target remote after a delete operation on the source - - Reduced storage efficiency on the target, as ``delete`` operations result in ``DeleteMarker`` + - Reduced storage efficiency on the target, as ``delete`` operations result in ``DeleteMarker`` rather than freeing space - Duplicate delete markers on source and target buckets -- The target bucket *can* have its own set of object management rules. - - Target buckets should *not* have their own rules for expiration or additional tiering. - MinIO supports only one level of tiering. - Mismatched rules between the source and the target buckets run a risk of data corruption and loss. - Exclusive Access to Remote Data ------------------------------- From 1c7afccb50cb3e23e525aba02feb61e283ca094c Mon Sep 17 00:00:00 2001 From: Daryl White <53910321+djwfyi@users.noreply.github.com> Date: Wed, 22 May 2024 13:57:11 -0400 Subject: [PATCH 5/5] Final PR fixes --- source/administration/object-management.rst | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/source/administration/object-management.rst b/source/administration/object-management.rst index 60ac016e..58e3bf0e 100644 --- a/source/administration/object-management.rst +++ b/source/administration/object-management.rst @@ -231,19 +231,13 @@ Expiration rules can result in removal of tiered data still in use by the source Tiering to an additional remote creates an additional network hop between the hot tier and it's data while also increasing operational complexity. You *may* configure object locking or versioning on the remote bucket. -Use caution, as mismatched rules between the source and the target buckets run a risk of data corruption and loss. -Creating separate rules on the target bucket may have effects such as the following: +Enabling versioning or object locking on the target bucket may have effects such as the following: - Object locking set on the target bucket may prevent desired ``delete`` operations from the source bucket from completing. - -- MinIO tiers objects with their own ``UUID``, so versioning on the target bucket is not required. - - If you enable versioning on the target bucket, you may experience the following unexpected or undesired results: - - - Excess data usage on the target remote after a delete operation on the source - - Reduced storage efficiency on the target, as ``delete`` operations result in ``DeleteMarker`` rather than freeing space - - Duplicate delete markers on source and target buckets +- MinIO tiers objects with their own ``UUID``, so versioning on the target bucket is redundant at best. +- Reduced storage efficiency on the target, as ``delete`` operations result in creation of a ``DeleteMarker`` rather than freeing space. +- Duplicate delete markers on source and target buckets. Exclusive Access to Remote Data -------------------------------