From 23253dd71cf3f0de61caa85a85d25f453fa48d96 Mon Sep 17 00:00:00 2001 From: Ravind Kumar Date: Thu, 1 Aug 2024 15:51:39 -0400 Subject: [PATCH] DOCS-1273: Operator 6.0.0 Deploy/Upgrade docs, removing Console references (#1284) Addresses #1273 # Summary This pass does three things: 1. Updates all tutorials related to Operator/Tenant deployment for Kustomize and Helm 2. Removes references to Operator Console + updates to reference Kustomize/Helm wherever possible 3. Slightly tidies up old or dangling references This pass does not do these things: - Link out heavily to Kubernetes docs (for later) - Clean up organization (singleplat build handles this) - Addresses OpenShift, Rancher, etc. --------- Co-authored-by: Andrea Longo Co-authored-by: Allan Roger Reid Co-authored-by: Daryl White <53910321+djwfyi@users.noreply.github.com> --- Makefile | 3 + source/_static/scss/includes/_misc.scss | 12 + .../console/managing-deployment.rst | 8 +- source/default-conf.py | 4 +- .../common-install-operator-kustomize.rst | 100 +--- ...k8s-connect-operator-console-no-plugin.rst | 2 - source/includes/k8s/deploy-operator.rst | 36 +- ...configure-keycloak-identity-management.rst | 48 -- .../openshift/deploy-minio-on-openshift.rst | 25 +- source/operations/concepts.rst | 3 +- source/operations/deploy-manage-tenants.rst | 1 - .../delete-minio-tenant.rst | 39 +- .../deploy-minio-tenant-helm.rst | 311 +++++++--- .../deploy-minio-tenant.rst | 536 ++++-------------- .../deploy-operator-helm.rst | 389 +------------ .../expand-minio-tenant.rst | 103 +++- .../minio-operator-console.rst | 10 + .../modify-minio-tenant.rst | 90 +-- .../upgrade-minio-operator-4.5.7-earlier.rst | 291 +++++++++- .../upgrade-minio-operator.rst | 316 ++++------- 20 files changed, 955 insertions(+), 1372 deletions(-) diff --git a/Makefile b/Makefile index d007d505..8339c9e0 100644 --- a/Makefile +++ b/Makefile @@ -174,6 +174,7 @@ sync-operator-version: @echo "Retrieving latest Operator version" @$(eval OPERATOR = $(shell curl --retry 10 -Ls -o /dev/null -w "%{url_effective}" https://github.com/minio/operator/releases/latest | sed "s/https:\/\/github.com\/minio\/operator\/releases\/tag\///" | sed "s/v//")) @$(eval kname = $(shell uname -s)) + @$(eval K8SFLOOR = $(shell curl -sL https://raw.githubusercontent.com/minio/operator/master/testing/kind-config-floor.yaml | grep -F -m 1 'node:v' | awk 'BEGIN { FS = ":" } ; {print $$3}')) @echo "Updating Operator to ${OPERATOR}" @@ -182,9 +183,11 @@ sync-operator-version: @case "${kname}" in \ "Darwin") \ sed -i "" "s|OPERATOR|${OPERATOR}|g" source/conf.py;\ + sed -i "" "s|K8SFLOOR|${K8SFLOOR}|g" source/conf.py; \ ;; \ *) \ sed -i "s|OPERATOR|${OPERATOR}|g" source/conf.py; \ + sed -i "s|K8SFLOOR|${K8SFLOOR}|g" source/conf.py; \ ;; \ esac diff --git a/source/_static/scss/includes/_misc.scss b/source/_static/scss/includes/_misc.scss index 5422e37b..e0ff5821 100644 --- a/source/_static/scss/includes/_misc.scss +++ b/source/_static/scss/includes/_misc.scss @@ -379,4 +379,16 @@ figcaption { table thead tr.header { background-color: lightgray; box-shadow: none; +} + +// -------------------------------------- +// Improving readability of ordered lists +// -------------------------------------- + +div.procedure ol.arabic li > p:first-child { + font-weight: bold; +} + +div.procedure ol.arabic li ul.simple li p { + font-weight: normal; // stupid hack, not sure how else to fix this } \ No newline at end of file diff --git a/source/administration/console/managing-deployment.rst b/source/administration/console/managing-deployment.rst index c3d0f1e0..d3621792 100644 --- a/source/administration/console/managing-deployment.rst +++ b/source/administration/console/managing-deployment.rst @@ -18,6 +18,12 @@ You can use the MinIO Console to perform many of the deployment monitoring and m - Set up :ref:`site replication ` to synchronize datacenters for timely access across geographically dispersed workforces or for disaster preparedness. - Configure deployment :ref:`settings `. +.. important:: + + The MinIO Console is a web-based interface for the MinIO Server. + + It is distinct and independent from the MinIO Kubernetes Operator Console which was deprecated and removed as of Operator 6.0.0. + .. _minio-console-monitoring: Monitoring @@ -62,8 +68,6 @@ Audit The Audit Log section provides an interface for viewing :ref:`audit logs ` collected by a configured PostgreSQL service. -The Audit Logging feature is configured and enabled automatically for MinIO deployments created using the :ref:`MinIO Operator Console `. - Trace ~~~~~ diff --git a/source/default-conf.py b/source/default-conf.py index 516e3a02..c437f059 100644 --- a/source/default-conf.py +++ b/source/default-conf.py @@ -331,10 +331,12 @@ .. |SNSD| replace:: :abbr:`SNSD (Single-Node Single-Drive)` .. |SNMD| replace:: :abbr:`SNMD (Single-Node Multi-Drive)` .. |MNMD| replace:: :abbr:`MNMD (Multi-Node Multi-Drive)` -.. |operator-version-stable| replace:: 5.0.15 + +.. |operator-version-stable| replace:: OPERATOR .. |helm-charts| replace:: `Helm Charts `__ .. |helm-operator-chart| replace:: `Helm Operator Charts `__ .. |helm-tenant-chart| replace:: `Helm Tenant Charts `__ +.. |k8s-floor| replace:: K8SFLOOR .. |cpp-sdk-version| replace:: CPPVERSION .. |dotnet-sdk-version| replace:: DOTNETVERSION diff --git a/source/includes/common/common-install-operator-kustomize.rst b/source/includes/common/common-install-operator-kustomize.rst index 75ef08ee..6babe6e2 100644 --- a/source/includes/common/common-install-operator-kustomize.rst +++ b/source/includes/common/common-install-operator-kustomize.rst @@ -66,42 +66,6 @@ The following procedure uses ``kubectl -k`` to install the Operator from the Min .. _minio-k8s-deploy-operator-access-console: -#. *(Optional)* Configure access to the Operator Console service - - The Operator Console service does not automatically bind or expose itself for external access on the Kubernetes cluster. - You must instead configure a network control plane component, such as a load balancer or ingress, to grant that external access. - - For testing purposes or short-term access, expose the Operator Console service through a NodePort using the following patch: - - .. code-block:: shell - :class: copyable - - kubectl patch service -n minio-operator console -p ' - { - "spec": { - "ports": [ - { - "name": "http", - "port": 9090, - "protocol": "TCP", - "targetPort": 9090, - "nodePort": 30090 - }, - { - "name": "https", - "port": 9443, - "protocol": "TCP", - "targetPort": 9443, - "nodePort": 30433 - } - ], - "type": "NodePort" - } - }' - - The patch command should output ``service/console patched``. - You can now access the service through ports ``30433`` (HTTPS) or ``30090`` (HTTP) on any of your Kubernetes worker nodes. - #. Verify the Operator installation Check the contents of the specified namespace (``minio-operator``) to ensure all pods and services have started successfully. @@ -123,7 +87,6 @@ The following procedure uses ``kubectl -k`` to install the Operator from the Min NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/operator ClusterIP 10.43.135.241 4221/TCP 5m20s service/sts ClusterIP 10.43.117.251 4223/TCP 5m20s - service/console NodePort 10.43.235.38 9090:30090/TCP,9443:30433/TCP 5m20s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/console 1/1 1 1 5m20s @@ -133,63 +96,10 @@ The following procedure uses ``kubectl -k`` to install the Operator from the Min replicaset.apps/console-56c7d8bd89 1 1 1 5m20s replicaset.apps/minio-operator-6c758b8c45 2 2 2 5m20s -#. Retrieve the Operator Console JWT for login - - .. code-block:: shell - :class: copyable - - kubectl apply -f - <` + MinIO also provides a :ref:`Helm chart for deploying Tenants `. -Once you access the Console, use the Console JWT to log in. -You can now :ref:`deploy and manage MinIO Tenants using the Operator Console `. + MinIO recommends using the same method of Tenant deployment and management used to install the Operator. + Mixing Kustomize and Helm for Operator or Tenant management may increase operational complexity. diff --git a/source/includes/common/common-k8s-connect-operator-console-no-plugin.rst b/source/includes/common/common-k8s-connect-operator-console-no-plugin.rst index 64ba0fcc..64c2bff9 100644 --- a/source/includes/common/common-k8s-connect-operator-console-no-plugin.rst +++ b/source/includes/common/common-k8s-connect-operator-console-no-plugin.rst @@ -1,8 +1,6 @@ The Operator Console service does not automatically bind or expose itself for external access on the Kubernetes cluster. Instead, you must configure a network control plane component, such as a load balancer or ingress, to grant external access. -.. cond:: k8s - For testing purposes or short-term access, expose the Operator Console service through a NodePort using the following patch: .. code-block:: shell diff --git a/source/includes/k8s/deploy-operator.rst b/source/includes/k8s/deploy-operator.rst index 6a210937..0399c2b8 100644 --- a/source/includes/k8s/deploy-operator.rst +++ b/source/includes/k8s/deploy-operator.rst @@ -31,16 +31,15 @@ While this documentation *may* provide guidance for configuring or deploying Kub MinIO Operator Components ------------------------- -The MinIO Operator exists in its own namespace. -Within the Operator's namespace, the MinIO Operator utilizes two pods: +The MinIO Operator exists in its own namespace in which it creates Kubernetes resources. +Those resources includes pods, services, replicasets, and deployments. -- The Operator pod for the base Operator functions to deploy, manage, modify, and maintain tenants. -- Console pod for the Operator's Graphical User Interface, the Operator Console. +The Operator pods monitor all namespaces by default for objects using the MinIO CRD and manages those resources automatically. When you use the Operator to create a tenant, the tenant *must* have its own namespace. Within that namespace, the Operator generates the pods required by the tenant configuration. -Each pod runs three containers: +Each Tenant pod runs three containers: - MinIO Container that runs all of the standard MinIO functions, equivalent to basic MinIO installation on baremetal. This container stores and retrieves objects in the provided mount points (persistent volumes). @@ -51,32 +50,29 @@ Each pod runs three containers: - SideCar container that monitors configuration secrets for the tenant and updates them as they change. This container also monitors for root credentials and creates an error if it does not find root credentials. -Starting with v5.0.6, the MinIO Operator supports custom :kube-docs:`init containers ` for additional pod initialization that may be required for your environment. - The tenant utilizes Persistent Volume Claims to talk to the Persistent Volumes that store the objects. -.. image:: /images/k8s/OperatorsComponent-Diagram.png - :width: 600px - :alt: A diagram of the namespaces and pods used by or maintained by the MinIO Operator. - :align: center +.. Image references Console pods, need to fix this up + +.. .. image:: /images/k8s/OperatorsComponent-Diagram.png +.. :width: 600px +.. :alt: A diagram of the namespaces and pods used by or maintained by the MinIO Operator. +.. :align: center .. _minio-operator-prerequisites: Prerequisites ------------- -Kubernetes Version 1.21.0 -~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. important:: +Kubernetes Version |k8s-floor| +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - MinIO **strongly recommends** upgrading Production clusters running `End-Of-Life `__ Kubernetes APIs. +MinIO tests |operator-version-stable| against a floor of Kubernetes API of |k8s-floor|. +MinIO **strongly recommends** maintaining Kubernetes infrastructure using `actively maintained Kubernetes API versions `__. -Starting with v5.0.0, MinIO **requires** Kubernetes 1.21.0 or later for both the infrastructure and the ``kubectl`` CLI tool. -.. versionadded:: Operator 5.0.6 +MinIO **strongly recommends** upgrading Kubernetes clusters running with `End-Of-Life API versions `__ -For Kubernetes 1.25.0 and later, MinIO supports deploying in environments with the :kube-docs:`Pod Security admission (PSA) ` ``restricted`` policy enabled. Kustomize and ``kubectl`` @@ -100,7 +96,7 @@ Kubernetes TLS Certificate API The MinIO Operator manages TLS Certificate Signing Requests (CSR) using the Kubernetes ``certificates.k8s.io`` :kube-docs:`TLS certificate management API ` to create signed TLS certificates in the following circumstances: - When ``autoCert`` is enabled. - - For the MinIO Console when the :envvar:`MINIO_CONSOLE_TLS_ENABLE` environment variable is set to ``on``. + - For the MinIO Tenant Console when the :envvar:`MINIO_CONSOLE_TLS_ENABLE` environment variable is set to ``on``. - For :ref:`STS service ` when :envvar:`OPERATOR_STS_ENABLED` environment variable is set to ``on``. - For retrieving the health of the cluster. diff --git a/source/includes/k8s/steps-configure-keycloak-identity-management.rst b/source/includes/k8s/steps-configure-keycloak-identity-management.rst index 515babe3..bbe34929 100644 --- a/source/includes/k8s/steps-configure-keycloak-identity-management.rst +++ b/source/includes/k8s/steps-configure-keycloak-identity-management.rst @@ -37,59 +37,11 @@ Set the value to any :ref:`policy ` on the MinIO deployment. MinIO supports multiple methods for configuring Keycloak authentication: -- Using the MinIO Operator Console - Using the MinIO Tenant Console - Using a terminal/shell and the :mc:`mc idp openid` command .. tab-set:: - .. tab-item:: MinIO Operator Console - - You can use the MinIO Operator Console to configure Keycloak as the External Identity Provider for the MinIO Tenant. - See :ref:`minio-operator-console-connect` for specific instructions. - - Select :guilabel:`Identity Provider` from the left-hand navigation bar, then select :guilabel:`OpenID`. - Select :guilabel:`Create Configuration` to create a new configuration. - - Enter the following information into the modal: - - .. list-table:: - :stub-columns: 1 - :widths: 30 70 - :width: 100% - - * - :guilabel:`Name` - - Enter a unique name for the Keycloak instance - - * - :guilabel:`Config URL` - - Specify the address of the Keycloak OpenID configuration document (|KEYCLOAK_URL|) - - Ensure the ``REALM`` matches the Keycloak realm you want to use for authenticating users to MinIO - - * - :guilabel:`Client ID` - - Specify the name of the Keycloak client created in Step 1 - - * - :guilabel:`Client Secret` - - Specify the secret credential value for the Keycloak client created in Step 1 - - * - :guilabel:`Display Name` - - Specify the user-facing name the MinIO Console should display as part of the Single-Sign On (SSO) workflow for the configured Keycloak service - - * - :guilabel:`Scopes` - - Specify the OpenID scopes to include in the JWT, such as ``preferred_username`` or ``email`` - - You can reference these scopes using supported OpenID policy variables for the purpose of programmatic policy configurations - - * - :guilabel:`Redirect URI Dynamic` - - Toggle to ``on`` - - Substitutes the MinIO Console address used by the client as part of the Keycloak redirect URI. - Keycloak returns authenticated users to the Console using the provided URI. - - For MinIO Console deployments behind a reverse proxy, load balancer, or similar network control plane, you can instead use the :envvar:`MINIO_BROWSER_REDIRECT_URL` variable to set the redirect address for Keycloak to use. - - Select :guilabel:`Save` to apply the configuration. - .. tab-item:: MinIO Tenant Console You can use the MinIO Tenant Console to configure Keycloak as the External Identity Provider for the MinIO Tenant. diff --git a/source/includes/openshift/deploy-minio-on-openshift.rst b/source/includes/openshift/deploy-minio-on-openshift.rst index 9af77bab..13bb7885 100644 --- a/source/includes/openshift/deploy-minio-on-openshift.rst +++ b/source/includes/openshift/deploy-minio-on-openshift.rst @@ -133,7 +133,7 @@ Once installation completes, click :guilabel:`View Operator` to view the MinIO O ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you have installed the MinIO Operator from Red Hat OperatorHub, the installation process also configures the :openshift-docs:`OpenShift Service CA Operator `. -This Operator manages the TLS certificates required to access the MinIO Operator Console and Tenants. +This Operator manages the TLS certificates required to access MinIO Tenants. It automatically renews and rotates the certificates 13 months before expiration. No additional action is required. @@ -287,29 +287,8 @@ You can find the MinIO Operator Interface from the :guilabel:`Operators` left-ha 3. Select :guilabel:`MinIO Operators` from the list of installed operators. The :guilabel:`Status` column must read :guilabel:`Success` to access the Operator interface. -5) Access the Operator Console -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The MinIO Operator includes the Operator Console, a browser-based management interface for managed MinIO tenants. - -.. include:: /includes/common/common-k8s-connect-operator-console.rst - -You can create a permanent routing rule by creating a :openshift-docs:`Route or Ingress ` to allow access from external clients, such as your local computer browser. - -The following steps provides a summary of actions necessary to create a Route. - -#. From :guilabel:`Networking`, go to :guilabel:`Routes` -#. Create a new Route in the MinIO Operator project. - Select a recognizable route name, such as ``operator-console-route``. -#. Set the :guilabel:`Hostname` as per your organizations networking and hostname topology. - Omit the hostname to allow OpenShift to generate it automatically -#. Set the :guilabel:`Service` to :guilabel:`console` -#. Set the :Guilabel:`Target Port` to ``9090`` - -You can then access the Operator Console using the configured Route. -The Operator Console still requires using the generated JWT token for access, which you can generate at any time using ``oc minio port-forward``. - -6) Next Steps +5) Next Steps ~~~~~~~~~~~~~ After deploying the MinIO Operator, you can create a new MinIO Tenant. diff --git a/source/operations/concepts.rst b/source/operations/concepts.rst index 9890b7d0..45f0ce7b 100644 --- a/source/operations/concepts.rst +++ b/source/operations/concepts.rst @@ -132,7 +132,8 @@ There are several options to manage your MinIO deployments and clusters: - Use the command line with :mc:`mc` and :mc:`mc admin` - The :ref:`MinIO Console ` graphical user interface for individual instances -- In Kubernetes, with the :ref:`MinIO Operator Console ` + +.. Reference Enterprise Operator Console eventually .. _minio-rebalance: diff --git a/source/operations/deploy-manage-tenants.rst b/source/operations/deploy-manage-tenants.rst index 3aabb78f..ed8ec7a7 100644 --- a/source/operations/deploy-manage-tenants.rst +++ b/source/operations/deploy-manage-tenants.rst @@ -48,4 +48,3 @@ The following tutorials provide steps for tenant management via the Operator Con /operations/install-deploy-manage/expand-minio-tenant /operations/install-deploy-manage/delete-minio-tenant /operations/install-deploy-manage/multi-site-replication - /operations/install-deploy-manage/minio-operator-console diff --git a/source/operations/install-deploy-manage/delete-minio-tenant.rst b/source/operations/install-deploy-manage/delete-minio-tenant.rst index 74be7c23..8043913b 100644 --- a/source/operations/install-deploy-manage/delete-minio-tenant.rst +++ b/source/operations/install-deploy-manage/delete-minio-tenant.rst @@ -37,13 +37,36 @@ The delete behavior of each Persistent Volume Claims (``PVC``) generated by the Perform all due diligence in ensuring the safety of stored data *prior* to deleting the Tenant. -Procedure (Operator Console) ----------------------------- - -#. From the Operator Console view, select the Tenant to open the summary view, then select :guilabel:`Pools`. - Select :guilabel:`Delete`. +Procedure +--------- -#. Enter the Tenant name in the confirmation dialog to confirm you want to delete this Tenant. - To delete the Tenant's associated volumes, toggle :guilabel:`Delete Volumes` to :guilabel:`ON`. +.. tab-set:: -#. Select :guilabel:`Delete`. + .. tab-item:: Kustomization + + You can delete a Kustomization-installed Tenant by deleting the namespace: + + .. code-block:: shell + + kubectl delete namespace TENANT-NAMESPACE + + Replace ``TENANT-NAMESPACE`` with the name of the namespace to remove. + + .. important:: + + Ensure you have specified the correct namespace for removal before running the command. + Namespace removal occurs at the Kubernetes layer, such that the MinIO Operator cannot interfere with nor undo the operation. + + .. tab-item:: Helm + + You can delete a Helm-installed namespace by using the ``helm uninstall`` command: + + .. code-block:: shell + + helm uninstall --namespace MINIO-TENANT TENANT-NAME minio-operator/tenant + + The command above assumes use of the MinIO Operator Chart repository. + If you installed the Chart manually or by using a different repository name, specify that chart or name in the command. + + Replace ``TENANT-NAME`` and ``TENANT-NAMESPACE`` with the name and namespace of the Tenant respectively. + You can use ``helm list -n TENANT-NAMESPACE`` to validate the Tenant name. diff --git a/source/operations/install-deploy-manage/deploy-minio-tenant-helm.rst b/source/operations/install-deploy-manage/deploy-minio-tenant-helm.rst index b7373729..b9e527bc 100644 --- a/source/operations/install-deploy-manage/deploy-minio-tenant-helm.rst +++ b/source/operations/install-deploy-manage/deploy-minio-tenant-helm.rst @@ -22,7 +22,6 @@ You cannot use the MinIO Operator Tenant chart to deploy a Tenant independent of .. important:: - The MinIO Operator Tenant Chart is *distinct* from the community-managed :minio-git:`MinIO Chart `. The Community Helm Chart is built, maintained, and supported by the community. @@ -56,111 +55,192 @@ Deploy a MinIO Tenant using Helm Charts The following procedure deploys a MinIO Tenant using the MinIO Operator Chart Repository. This method supports a simplified installation path compared to the :ref:`local chart installation `. -You can modify the Operator deployment after installation. + + +The following procedure uses Helm to deploy a MinIO Tenant using the official MinIO Tenant Chart. .. important:: - Do not use the ``kubectl krew`` or similar methods to update or manage the MinIO Tenant installation. - If you use Helm charts to deploy the Tenant, you must use Helm to manage that deployment. + If you use Helm to deploy a MinIO Tenant, you must use Helm to manage or upgrade that deployment. + Do not use ``kubectl krew``, Kustomize, or similar methods to manage or upgrade the MinIO Tenant. -#. Validate the Operator Repository Contents +This procedure is not exhaustive of all possible configuration options available in the :ref:`Tenant Chart `. +It provides a baseline from which you can modify and tailor the Tenant to your requirements. - Use ``helm search`` to check the latest available chart version in the Operator Repo: +.. container:: procedure - .. code-block:: shell - :class: copyable + #. Verify your MinIO Operator Repo Configuration - helm search repo minio-operator + MinIO maintains a Helm-compatible repository at https://operator.min.io. + If the repository does not already exist in your local Helm configuration, add it before continuing: - The response should resemble the following: + .. code-block:: shell + :class: copyable - .. code-block:: shell - :class: copyable + helm repo add minio-operator https://operator.min.io - NAME CHART VERSION APP VERSION DESCRIPTION - minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator - minio-operator/operator 5.0.10 v5.0.10 A Helm chart for MinIO Operator - minio-operator/tenant 5.0.10 v5.0.10 A Helm chart for MinIO Operator + You can validate the repo contents using ``helm search``: - The ``minio-operator/minio-operator`` is a legacy chart and should **not** be installed under normal circumstances. + .. code-block:: shell + :class: copyable - If your ``minio-operator/operator`` version is behind the latest available chart, upgrade the operator *first*. + helm search repo minio-operator -#. Deploy the Helm Chart + The response should resemble the following: - Use the ``helm install`` command to deploy the Tenant Chart. + .. code-block:: shell + :class: copyable + :substitutions: - If you need to override values in the default :ref:`values ` file, you can use the ``--set`` operation for any single key-value. - Alternatively, specify your own ``values.yaml`` using the ``--f`` parameter to override multiple values at once: + NAME CHART VERSION APP VERSION DESCRIPTION + minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator + minio-operator/operator |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator + minio-operator/tenant |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator - .. code-block:: shell - :class: copyable + #. Create a local copy of the Helm ``values.yaml`` for modification - helm install \ - --namespace MINIO_TENANT_NAMESPACE \ - --create-namespace \ - MINIO_TENANT_NAME minio-operator/tenant + .. code-block:: shell + :class: copyable - For details on the options available in the MinIO Tenant ``values.yaml``, see :ref:`minio-tenant-chart-values`. + curl -sLo values.yaml https://raw.githubusercontent.com/minio/operator/master/helm/tenant/values.yaml + Open the ``values.yaml`` object in your preferred text editor. -#. Validate the Tenant installation + #. Configure the Tenant topology + + The following fields share the ``tenant.pools[0]`` prefix and control the number of servers, volumes per server, and storage class of all pods deployed in the Tenant: + + .. list-table:: + :header-rows: 1 + :widths: 30 70 - Check the contents of the specified namespace to ensure all pods and services have started successfully. + * - Field + - Description - .. code-block:: shell - :class: copyable + * - ``servers`` + - The number of MinIO pods to deploy in the Server Pool. + + * - ``volumesPerServer`` + - The number of persistent volumes to attach to each MinIO pod (``servers``). + The Operator generates ``volumesPerServer x servers`` Persistant Volume Claims for the Tenant. + + * - ``storageClassName`` + - The Kubernetes storage class to associate with the generated Persistent Volume Claims. - kubectl get all -n MINIO_TENANT_NAMESPACE + If no storage class exists matching the specified value *or* if the specified storage class cannot meet the requested number of PVCs or storage capacity, the Tenant may fail to start. - All pods and services should have a READY state before proceeding. + * - ``size`` + - The amount of storage to request for each generated PVC. -#. Expose the Tenant Console port + #. Configure Tenant Affinity or Anti-Affinity - Use ``kubectl port-forward`` to temporarily forward traffic from the MinIO pod to your local machine: + The Tenant Chart supports the following Kubernetes Selector, Affinity and Anti-Affinity configurations: - .. code-block:: shell - :class: copyable + - Node Selector (``tenant.nodeSelector``) + - Node/Pod Affinity or Anti-Affinity (``spec.pools[n].affinity``) - kubectl --namespace MINIO_TENANT_NAMESPACE port-forward svc/MINIO_TENANT_NAME-console 9443:9443 - - .. note:: - - To configure long term access to the pod, configure :kube-docs:`Ingress ` or similar network control components within Kubernetes to route traffic to and from the pod. - Configuring Ingress is out of the scope for this documentation. + MinIO recommends configuring Tenants with Pod Anti-Affinity to ensure that the Kubernetes schedule does not schedule multiple pods on the same worker node. -#. Login to the MinIO Console + If you have specific worker nodes on which you want to deploy the tenant, pass those node labels or filters to the ``nodeSelector`` or ``affinity`` field to constrain the scheduler to place pods on those nodes. - Access the Tenant's :ref:`minio-console` by navigating to ``http://localhost:9443`` in a browser. - Log in to the Console with the default credentials ``myminio | minio123``. - If you modified these credentials in the ``values.yaml`` specify those values instead. + #. Configure Network Encryption -#. Expose the Tenant MinIO S3 API port + The MinIO Tenant CRD provides the following fields with which you can configure tenant TLS network encryption: - To test the MinIO Client :mc:`mc` from your local machine, forward the MinIO port and create an alias. + .. list-table:: + :header-rows: 1 + :widths: 30 70 - * Forward the Tenant's MinIO port: + * - Field + - Description - .. code-block:: shell - :class: copyable + * - ``tenant.certificate.requestAutoCert`` + - Enable or disable MinIO :ref:`automatic TLS certificate generation `. - kubectl port-forward svc/MINIO_TENANT_NAME-hl 9000 -n MINIO_TENANT_NAMESPACE + Defaults to ``true`` or enabled if omitted. - * Create an alias for the Tenant service: + * - ``tenant.certificate.certConfig`` + - Customize the behavior of :ref:`automatic TLS `, if enabled. - .. code-block:: shell - :class: copyable + * - ``tenant.certificate.externalCertSecret`` + - Enable TLS for multiple hostnames via Server Name Indication (SNI). + + Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` or ``cert-manager``. - mc alias set myminio https://localhost:9000 minio minio123 --insecure + * - ``tenant.certificate.externalCACertSecret`` + - Enable validation of client TLS certificates signed by unknown, third-party, or internal Certificate Authorities (CA). + + Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` containing the full chain of CA certificates for a given authority. - You can use :mc:`mc mb` to create a bucket on the Tenant: - - .. code-block:: shell - :class: copyable + #. Configure MinIO Environment Variables + + You can set MinIO Server environment variables using the ``tenant.configuration`` field. + + .. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Field + - Description + + * - ``tenant.configuration`` + - Specify a Kubernetes opaque secret whose data payload ``config.env`` contains each MinIO environment variable you want to set. + + The ``config.env`` data payload **must** be a base64-encoded string. + You can create a local file, set your environment variables, and then use ``cat LOCALFILE | base64`` to create the payload. + + The YAML includes an object ``kind: Secret`` with ``metadata.name: storage-configuration`` that sets the root username, password, erasure parity settings, and enables Tenant Console. + + Modify this as needed to reflect your Tenant requirements. + + #. Deploy the Tenant + + Use ``helm`` to install the Tenant Chart using your ``values.yaml`` as an override: + + .. code-block:: shell + :class: copyable + + helm install \ + --namespace TENANT-NAMESPACE \ + --create-namespace \ + --values values.yaml \ + TENANT-NAME minio-operator/tenant + + You can monitor the progress using the following command: + + .. code-block:: shell + :class: copyable + + watch kubectl get all -n TENANT-NAMESPACE + + #. Expose the Tenant MinIO S3 API port + + To test the MinIO Client :mc:`mc` from your local machine, forward the MinIO port and create an alias. - mc mb myminio/mybucket --insecure + * Forward the Tenant's MinIO port: - If you deployed your MinIO Tenant using TLS certificates minted by a trusted Certificate Authority (CA) you can omit the ``--insecure`` flag. + .. code-block:: shell + :class: copyable + + kubectl port-forward svc/TENANT-NAME-hl 9000 -n TENANT-NAMESPACE + + * Create an alias for the Tenant service: + + .. code-block:: shell + :class: copyable + + mc alias set myminio https://localhost:9000 minio minio123 --insecure + + You can use :mc:`mc mb` to create a bucket on the Tenant: + + .. code-block:: shell + :class: copyable + + mc mb myminio/mybucket --insecure + + If you deployed your MinIO Tenant using TLS certificates minted by a trusted Certificate Authority (CA) you can omit the ``--insecure`` flag. + + See :ref:`create-tenant-connect-tenant` for additional documentation on external connectivity to the Tenant. .. _deploy-tenant-helm-local: @@ -182,8 +262,78 @@ This method may support easier pre-configuration of the Tenant compared to the : Each chart contains a ``values.yaml`` file you can customize to suit your needs. For details on the options available in the MinIO Tenant ``values.yaml``, see :ref:`minio-tenant-chart-values`. - For example, you may wish to change the MinIO root user credentials or the Tenant name. - For more about customizations, see `Helm Charts `__. + + Open the ``values.yaml`` object in your preferred text editor. + +#. Configure the Tenant topology + + The following fields share the ``tenant.pools[0]`` prefix and control the number of servers, volumes per server, and storage class of all pods deployed in the Tenant: + + .. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Field + - Description + + * - ``servers`` + - The number of MinIO pods to deploy in the Server Pool. + * - ``volumesPerServer`` + - The number of persistent volumes to attach to each MinIO pod (``servers``). + The Operator generates ``volumesPerServer x servers`` Persistant Volume Claims for the Tenant. + * - ``storageClassName`` + - The Kubernetes storage class to associate with the generated Persistent Volume Claims. + + If no storage class exists matching the specified value *or* if the specified storage class cannot meet the requested number of PVCs or storage capacity, the Tenant may fail to start. + + * - ``size`` + - The amount of storage to request for each generated PVC. + +#. Configure Tenant Affinity or Anti-Affinity + + The Tenant Chart supports the following Kubernetes Selector, Affinity and Anti-Affinity configurations: + + - Node Selector (``tenant.nodeSelector``) + - Node/Pod Affinity or Anti-Affinity (``spec.pools[n].affinity``) + + MinIO recommends configuring Tenants with Pod Anti-Affinity to ensure that the Kubernetes schedule does not schedule multiple pods on the same worker node. + + If you have specific worker nodes on which you want to deploy the tenant, pass those node labels or filters to the ``nodeSelector`` or ``affinity`` field to constrain the scheduler to place pods on those nodes. + +#. Configure Network Encryption + + The MinIO Tenant CRD provides the following fields from which you can configure tenant TLS network encryption: + + .. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Field + - Description + + * - ``tenant.certificate.requestAutoCert`` + - Enables or disables MinIO :ref:`automatic TLS certificate generation ` + + * - ``tenant.certificate.certConfig`` + - Controls the settings for :ref:`automatic TLS `. + Requires ``spec.requestAutoCert: true`` + + * - ``tenant.certificate.externalCertSecret`` + - Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` or ``cert-manager``. + MinIO uses these certificates for performing TLS handshakes based on hostname (Server Name Indication). + + * - ``tenant.certificate.externalCACertSecret`` + - Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` with the Certificate Authority (CA) chains which the Tenant must trust for allowing client TLS connections. + +#. Configure MinIO Environment Variables + + You can set MinIO Server environment variables using the ``tenant.configuration`` field. + + The field must specify a Kubernetes opaque secret whose data payload ``config.env`` contains each MinIO environment variable you want to set. + + The YAML includes an object ``kind: Secret`` with ``metadata.name: storage-configuration`` that sets the root username, password, erasure parity settings, and enables Tenant Console. + + Modify this as needed to reflect your Tenant requirements. #. The following Helm command creates a MinIO Tenant using the standard chart: @@ -192,32 +342,13 @@ This method may support easier pre-configuration of the Tenant compared to the : :substitutions: helm install \ - --namespace MINIO_TENANT_NAMESPACE \ + --namespace TENANT-NAMESPACE \ --create-namespace \ - MINIO_TENANT_NAME tenant-|operator-version-stable|.tgz + TENANT-NAME tenant-|operator-version-stable|.tgz To deploy more than one Tenant, create a Helm chart with the details of the new Tenant and repeat the deployment steps. Redeploying the same chart updates the previously deployed Tenant. -#. Expose the Tenant Console port - - Use ``kubectl port-forward`` to temporarily forward traffic from the MinIO pod to your local machine: - - .. code-block:: shell - :class: copyable - - kubectl --namespace MINIO_TENANT_NAMESPACE port-forward svc/myminio-console 9443:9443 - - .. note:: - - To configure long term access to the pod, configure :kube-docs:`Ingress ` or similar network control components within Kubernetes to route traffic to and from the pod. - Configuring Ingress is out of the scope for this documentation. - -#. Login to the MinIO Console - - Access the Tenant's :ref:`minio-console` by navigating to ``http://localhost:9443`` in a browser. - Log in to the Console with the default credentials ``myminio | minio123``. - #. Expose the Tenant MinIO port To test the MinIO Client :mc:`mc` from your local machine, forward the MinIO port and create an alias. @@ -227,7 +358,7 @@ This method may support easier pre-configuration of the Tenant compared to the : .. code-block:: shell :class: copyable - kubectl port-forward svc/MINIO_TENANT_NAME-hl 9000 -n MINIO_TENANT_NAMESPACE + kubectl port-forward svc/TENANT-NAME-hl 9000 -n TENANT-NAMESPACE * Create an alias for the Tenant service: @@ -246,3 +377,5 @@ This method may support easier pre-configuration of the Tenant compared to the : :class: copyable mc mb myminio/mybucket --insecure + +See :ref:`create-tenant-connect-tenant` for additional documentation on external connectivity to the Tenant. \ No newline at end of file diff --git a/source/operations/install-deploy-manage/deploy-minio-tenant.rst b/source/operations/install-deploy-manage/deploy-minio-tenant.rst index fa8d2aea..e0f19b1f 100644 --- a/source/operations/install-deploy-manage/deploy-minio-tenant.rst +++ b/source/operations/install-deploy-manage/deploy-minio-tenant.rst @@ -25,7 +25,7 @@ Deploy a MinIO Tenant .. cond:: k8s and not openshift - This procedure documents deploying a MinIO Tenant onto a stock Kubernetes cluster using the MinIO Operator Console. + This procedure documents deploying a MinIO Tenant onto a stock Kubernetes cluster using either Kustomize or MinIO's Helm Charts. .. screenshot temporarily removed @@ -36,16 +36,10 @@ Deploy a MinIO Tenant :alt: MinIO Operator Console -The MinIO Operator Console is designed for deploying multi-node distributed MinIO Deployments. - Deploying Single-Node topologies requires additional configurations not covered in this documentation. You can alternatively use a simple Kubernetes YAML object to describe a Single-Node topology for local testing and evaluation as necessary. - MinIO does not recommend nor support single-node deployment topologies for production environments. -The Operator Console provides a rich user interface for deploying and managing MinIO Tenants on Kubernetes infrastructure. -Installing the MinIO :ref:`Kubernetes Operator ` automatically installs and configures the Operator Console. - This documentation assumes familiarity with all referenced Kubernetes concepts, utilities, and procedures. While this documentation *may* provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official :kube-docs:`Kubernetes Documentation <>`. @@ -64,15 +58,15 @@ See :ref:`deploy-operator-kubernetes` for complete documentation on deploying th .. cond:: k8s and not (openshift or eks or gke or aks) - Kubernetes Version 1.21.0 - ~~~~~~~~~~~~~~~~~~~~~~~~~ + Kubernetes Version |k8s-floor| + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - MinIO Operator requires Kubernetes 1.21.0 or later. - The Kubernetes infrastructure *and* the ``kubectl`` CLI tool must be the same version. - Upgrade ``kubectl`` to the same version as the Kubernetes version used on the cluster. + MinIO tests |operator-version-stable| against a floor of Kubernetes API of |k8s-floor|. + MinIO **strongly recommends** maintaining Kubernetes infrastructure using `actively maintained Kubernetes API versions `__. + + + MinIO **strongly recommends** upgrading Kubernetes clusters running with `End-Of-Life API versions `__. - This procedure assumes the host machine has ``kubectl`` installed and configured with access to the target Kubernetes cluster. - The host machine *must* have access to a web browser application. .. cond:: openshift @@ -175,6 +169,9 @@ Persistent Volumes MinIO can use any Kubernetes :kube-docs:`Persistent Volume (PV) ` that supports the :kube-docs:`ReadWriteOnce ` access mode. MinIO's consistency guarantees require the exclusive storage access that ``ReadWriteOnce`` provides. + The Persistent Volume **must** exist prior to deploying the Tenant. + + Additionally, MinIO recommends setting a reclaim policy of ``Retain`` for the PVC :kube-docs:`StorageClass `. Where possible, configure the Storage Class, CSI, or other provisioner underlying the PV to format volumes as XFS to ensure best performance. @@ -204,463 +201,185 @@ Persistent Volumes For more information on AKS disk types, see :azure-docs:`Azure disk types `. -Deploy a Tenant using the MinIO Operator Console ------------------------------------------------- - -To deploy a tenant from the MinIO Operator Console, complete the following steps in order: - -:ref:`create-tenant-access-minio-operator-console` - -:ref:`create-tenant-complete-tenant-setup` - -:ref:`create-tenant-configure-section` - -:ref:`create-tenant-images-section` - -:ref:`create-tenant-pod-placement-section` - -:ref:`create-tenant-identity-provider-section` - -:ref:`create-tenant-security-section` - -:ref:`create-tenant-encryption-section` - -:ref:`create-tenant-deploy-view-tenant` - -:ref:`create-tenant-connect-tenant` - -.. _create-tenant-access-minio-operator-console: - -1) Access the MinIO Operator Console -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. include:: /includes/common/common-k8s-connect-operator-console.rst - -Open your browser to the appropriate URL and enter the JWT Token into the login page. -You should see the :guilabel:`Tenants` page. - -.. screenshot temporarily removed - - .. image:: /images/k8s/operator-dashboard.png - :align: center - :width: 70% - :class: no-scaled-link - :alt: MinIO Operator Console - -Select :guilabel:`+ Create Tenant` to start creating a MinIO Tenant. - -.. _create-tenant-complete-tenant-setup: - -2) Complete the Tenant :guilabel:`Setup` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The :guilabel:`Setup` pane displays core configuration settings for the MinIO Tenant. - -Settings marked with an asterisk :guilabel:`*` are *required*: - -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% - - * - Field - - Description - - * - :guilabel:`Name` - - The name of the MinIO Tenant - - * - :guilabel:`Namespace` - - The Kubernetes Namespace in which to deploy the tenant. - You can create the namespace by selecting the plus :guilabel:`+` icon if it does not exist. - - The Operator supports at most *one* MinIO Tenant per namespace. - - * - :guilabel:`Storage Class` - - .. cond:: not eks - - Specify the Kubernetes Storage Class the Operator uses when generating Persistent Volume Claims for the Tenant. - - Ensure the specified storage class has sufficient available Persistent Volume resources to match each generated Persistent Volume Claim. - - .. cond:: eks - - Specify the EBS volume type to use for this tenant. - The following list is populated based on the AWS EBS CSI driver list of supported :github:`EBS volume types `: - - - ``gp3`` (General Purpose SSD) - - ``gp2`` (General Purpose SSD) - - ``io2`` (Provisioned IOPS SSD) - - ``io1`` (Provisioned IOPS SSD) - - ``st1`` (Throughput Optimized HDD) - - ``sc1`` (Cold Storage HDD) - - .. cond:: gke - - Specify the GKE persistent disk type to use for this tenant. - The :gke-docs:`GKE CSI Driver ` provides the following storage classes by default: - - - ``standard-rwo`` (Balanced Persistent SSD) - - ``premium-rwo`` (Performance Persistent SSD) - - You can create additional StorageClasses to represent other supported persistent disk types. - See :gke-docs:`Create a Storage Class ` for more information. - - .. cond:: aks - - Specify the AKS persistent disk type to use for this tenant. - The :aks-docs:`AKS CSI Driver ` provides the following storage classes by default: +Deploy a MinIO Tenant using Kustomize +------------------------------------- - - ``managed-csi`` (Standard SSD) - - ``managed-csi-premium`` (Premium SSD) +The following procedure uses ``kubectl -k`` to deploy a MinIO Tenant using the ``base`` Kustomization template in the :minio-git:`MinIO Operator Github repository `. - You can create additional Storage Classes to represent other supported persistent disk types. - See :aks-docs:`Create a custom storage class ` for more information. +You can select a different base or pre-built template from the :minio-git:`repository ` as your starting point, or build your own Kustomization resources using the :ref:`MinIO Custom Resource Documentation `. - * - :guilabel:`Number of Servers` - - The total number of MinIO server pods to deploy in the Tenant. - The Operator enforces a minimum of four server pods per tenant. - - The Operator by default uses pod anti-affinity, such that the Kubernetes cluster *must* have at least one worker node per MinIO server pod. - Use the :guilabel:`Pod Placement` pane to modify the pod scheduling settings for the Tenant. +.. important:: - * - :guilabel:`Drives per Server` - - The number of storage volumes (Persistent Volume Claims) the Operator requests per Server. + If you use Kustomize to deploy a MinIO Tenant, you must use Kustomize to manage or upgrade that deployment. + Do not use ``kubectl krew``, a Helm Chart, or similar methods to manage or upgrade the MinIO Tenant. - The Operator displays the :guilabel:`Total Volumes` under the :guilabel:`Resource Allocation` section. - The Operator generates an equal number of PVC *plus two* for supporting Tenant services (Metrics and Log Search). +This procedure is not exhaustive of all possible configuration options available in the :ref:`Tenant CRD `. +It provides a baseline from which you can modify and tailor the Tenant to your requirements. - .. cond:: not eks - - The specified :guilabel:`Storage Class` *must* correspond to a set of Persistent Volumes sufficient in number to match each generated PVC. +.. container:: procedure - .. cond:: eks + #. Create a YAML object for the Tenant - For deployments using the EBS CSI driver, the Operator provisions Persistent Volume Claims which result in the creation of EBS volumes of the specified :guilabel:`Storage Class` equal to ``Number of Drives per Server X Number of Servers``. + Use the ``kubectl kustomize`` command to produce a YAML file containing all Kubernetes resources necessary to deploy the ``base`` Tenant: - .. cond:: gke + .. code-block:: shell + :class: copyable - For deployments using the GKE CSI driver, the Operator provisions Persistent Volume Claims which result in the creation of GCP Disks of the specified :guilabel:`Storage Class` equal to ``Number of Drives per Server X Number of Servers``. + kubectl kustomize https://github.com/minio/operator/examples/kustomization/base/ > tenant-base.yaml - .. cond:: aks + The command creates a single YAML file with multiple objects separated by the ``---`` line. + Open the file in your preferred editor. - For deployments using the AKS CSI Driver, the Operator provisions Persistent Volume Claims which result in the creation of Azure Disks of the specified :guilabel:`Storage Class` equal to ``Number of Drives per Server X Number of Servers``. + The following steps reference each object based on it's ``kind`` and ``metadata.name`` fields: - * - :guilabel:`Total Size` - - The total raw storage size for the Tenant. - Specify both the total storage size *and* the :guilabel:`Unit` of that storage. - All storage units are in SI values, e.g. :math:`Gi = GiB = 1024^3` bytes. + #. Configure the Tenant topology - The Operator displays the :guilabel:`Drive Capacity` under the:guilabel:`Resource Allocation` section. - The Operator sets this value as the requested storage capacity in each generated PVC. + The ``kind: Tenant`` object describes the MinIO Tenant. - The specified :guilabel:`Storage Class` *must* correspond to a set of Persistent Volumes sufficient in capacity to match each generated PVC. + The following fields share the ``spec.pools[0]`` prefix and control the number of servers, volumes per server, and storage class of all pods deployed in the Tenant: + + .. list-table:: + :header-rows: 1 + :widths: 30 70 - * - :guilabel:`Erasure Code Parity` - - The Erasure Code Parity to set for the deployment. + * - Field + - Description - The Operator displays the selected parity and its effect on the deployment under the :guilabel:`Erasure Code Configuration` section. - Erasure Code parity defines the overall resiliency and availability of data on the cluster. - Higher parity values increase tolerance to drive or node failure at the cost of total storage. - See :ref:`minio-erasure-coding` for more complete documentation. + * - ``servers`` + - The number of MinIO pods to deploy in the Server Pool. - * - :guilabel:`CPU Request` - - Specify the desired number of CPUs to allocate per MinIO server pod. + * - ``volumesPerServer`` + - The number of persistent volumes to attach to each MinIO pod (``servers``). + The Operator generates ``volumesPerServer x servers`` Persistant Volume Claims for the Tenant. - * - :guilabel:`Memory Request [Gi]` - - Specify the desired amount of memory (RAM) to allocate per MinIO server pod. - See :ref:`minio-hardware-checklist-memory` for guidance on setting this value. - MinIO **requires** a minimum of 2GiB of memory per worker. + * - ``volumeClaimTemplate.spec.storageClassName`` + - The Kubernetes storage class to associate with the generated Persistent Volume Claims. - The Kubernetes cluster *must* have worker nodes with sufficient free RAM to match the pod request. + If no storage class exists matching the specified value *or* if the specified storage class cannot meet the requested number of PVCs or storage capacity, the Tenant may fail to start. - * - :guilabel:`Specify Limit` - - Toggle to :guilabel:`ON` to specify maximum CPU and memory limits. + * - ``volumeClaimTemplate.spec.resources.requests.storage`` + - The amount of storage to request for each generated PVC. -Select :guilabel:`Create` to create the Tenant using the current configuration. -While all subsequent sections are *optional*, MinIO recommends reviewing them prior to deploying the Tenant. + #. Configure Tenant Affinity or Anti-Affinity -.. _create-tenant-configure-section: + The MinIO Operator supports the following Kubernetes Affinity and Anti-Affinity configurations: -3) The :guilabel:`Configure` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + - Node Affinity (``spec.pools[n].nodeAffinity``) + - Pod Affinity (``spec.pools[n].podAffinity``) + - Pod Anti-Affinity (``spec.pools[n].podAntiAffinity``) -The :guilabel:`Configure` section displays optional configuration settings for the MinIO Tenant and its supporting services. + MinIO recommends configuring Tenants with Pod Anti-Affinity to ensure that the Kubernetes schedule does not schedule multiple pods on the same worker node. -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% + If you have specific worker nodes on which you want to deploy the tenant, pass those node labels or filters to the ``nodeAffinity`` field to constrain the scheduler to place pods on those nodes. - * - Field - - Description + #. Configure Network Encryption - * - :guilabel:`Expose MinIO Service` - - The MinIO Operator by default directs the MinIO Tenant services to request an externally accessible IP address from the Kubernetes cluster Load Balancer if one is available to access the tenant. + The MinIO Tenant CRD provides the following fields from which you can configure tenant TLS network encryption: - .. cond:: eks + .. list-table:: + :header-rows: 1 + :widths: 30 70 - If your EKS cluster includes the :aws-docs:`AWS Load Balancer Controller add-on `, enabling this setting directs the load balancer to assign an address to the Tenant services. + * - Field + - Description - Other load balancers *may* function similarly depending on their configuration. + * - ``tenant.certificate.requestAutoCert`` + - Enable or disable MinIO :ref:`automatic TLS certificate generation ` - .. cond:: not eks + Defaults to ``true`` or enabled if omitted. - Your Kubernetes distributions *may* include a load balancer that can respond to these requests. - Installation and configuration of load balancers is out of the scope of this documentation. + * - ``tenant.certificate.certConfig`` + - Customize the behavior of :ref:`automatic TLS `, if enabled. - * - :guilabel:`Expose Console Service` - - Select whether the Tenant should request an IP address from the Load Balancer to access the Tenant's Console. + * - ``tenant.certificate.externalCertSecret`` + - Enable TLS for multiple hostnames via Server Name Indication (SNI) + + Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` or ``cert-manager``. - .. cond:: eks + * - ``tenant.certificate.externalCACertSecret`` + - Enable validation of client TLS certificates signed by unknown, third-party, or internal Certificate Authorities (CA). + + Specify one or more Kubernetes secrets of type ``kubernetes.io/tls`` containing the full chain of CA certificates for a given authority. - If your EKS cluster includes the :aws-docs:`AWS Load Balancer Controller add-on `, enabling this setting directs the load balancer to assign an address to the Tenant services. + #. Configure MinIO Environment Variables - Other load balancers *may* function similarly depending on their configuration. + You can set MinIO Server environment variables using the ``tenant.configuration`` field. - .. cond:: not eks + .. list-table:: + :header-rows: 1 + :widths: 30 70 - Your Kubernetes distributions *may* include a load balancer that can respond to these requests. - Installation and configuration of load balancers is out of the scope of this documentation. + * - Field + - Description - * - :guilabel:`Set Custom Domains` - - Toggle on to customize the domains allowed to access the tenant's console and other tenant services. + * - ``tenant.configuration`` + - Specify a Kubernetes opaque secret whose data payload ``config.env`` contains each MinIO environment variable you want to set. - * - :guilabel:`Security Context` - - The MinIO Operator sets the Kubernetes Security Context for pods to a default of ``1000`` for User, Group, and FsGroup. - The FSGroupChangePolicy defaults to ``Always``. - MinIO does not run the pod using the ``root`` user. + The ``config.env`` data payload **must** be a base64-encoded string. + You can create a local file, set your environment variables, and then use ``cat LOCALFILE | base64`` to create the payload. - You can modify the Security Context to direct MinIO to run using a different User, Group,FsGroup ID, and FSGroupChangePolicy. - You can also direct MinIO to run as the Root user. + The YAML includes an object ``kind: Secret`` with ``metadata.name: storage-configuration`` that sets the root username, password, erasure parity settings, and enables Tenant Console. - .. cond:: openshift + Modify this as needed to reflect your Tenant requirements. - .. important:: + #. Review the Namespace - If your OpenShift cluster enforces :openshift-docs:`Security Context Constraints ` , ensure you set the Tenant constraints appropriately such that pods can start and run normally. + The YAML object ``kind: Namespace`` sets the default namespace for the Tenant to ``minio-tenant``. - * - :guilabel:`Custom Runtime Configurations` - - Toggle on to customize the :kube-docs:`Runtime Class ` for the tenant to use. + You can change this value to create a different namespace for the Tenant. + You must change **all** ``metadata.namespace`` values in the YAML file to match the Namespace. - * - :guilabel:`Additional Environment Variables` - - Enter any additional the key:value pairs to use as environment variables for the tenant. + #. Deploy the Tenant + Use the ``kubectl apply -f`` command to deploy the Tenant. -.. _create-tenant-images-section: + .. code-block:: shell + :class: copyable -4) The :guilabel:`Images` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + kubectl apply -f tenant-base.yaml -The :guilabel:`Images` section displays container image settings used by the MinIO Tenant. + The command creates each of the resources specified in the YAML object at the configured namespace. -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% + You can monitor the progress using the following command: - * - Field - - Description + .. code-block:: shell + :class: copyable - * - :guilabel:`MinIO` - - The container image to use for the MinIO Server. - See the `MinIO Quay `__ or the `MinIO DockerHub `__ repositories for a list of valid tags. + watch kubectl get all -n minio-tenant - * - :guilabel:`KES Image` - - The container image to use for MinIO :minio-git:`KES `. + #. Expose the Tenant MinIO S3 API port - * - :guilabel:`Use a private container registry` - - If the tenant requires a private container registry, toggle to :guilabel:`ON`, then specify the location and credentials for the private registry. + To test the MinIO Client :mc:`mc` from your local machine, forward the MinIO port and create an alias. -.. _create-tenant-pod-placement-section: + * Forward the Tenant's MinIO port: -5) The :guilabel:`Pod Placement` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + .. code-block:: shell + :class: copyable -The :guilabel:`Pod Placement` section displays pod scheduler settings for the MinIO Tenant. + kubectl port-forward svc/MINIO_TENANT_NAME-hl 9000 -n MINIO_TENANT_NAMESPACE -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% + * Create an alias for the Tenant service: - * - Field - - Description + .. code-block:: shell + :class: copyable - * - :guilabel:`None` - - Disables pod scheduling constraints for the tenant. - This allows Kubernetes to schedule multiple Tenant pods onto the same node. + mc alias set myminio https://localhost:9000 minio minio123 --insecure - This may decrease resiliency, as a single Kubernetes worker can host multiple MinIO pods. - If that worker is down or lost, objects may also be unavailable or lost. + You can use :mc:`mc mb` to create a bucket on the Tenant: + + .. code-block:: shell + :class: copyable - Consider using this setting only in early development or sandbox environments with a limited number of worker nodes. + mc mb myminio/mybucket --insecure - * - :guilabel:`Default (Pod Anti-Affinity)` - - Directs the Operator to set anti-affinity settings such that no Kubernetes worker can host more than one MinIO server pod for this Tenant. - - * - :guilabel:`Node Selector` - - Directs the operator to set a Node Selector such that pods only deploy onto Kubernetes workers whose labels match the selector. - - * - :guilabel:`Tolerations` - - Specify any required tolerations for this tenant's pods. - -.. _create-tenant-identity-provider-section: - -6) The :guilabel:`Identity Provider` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The :guilabel:`Identity Provider` section displays the :ref:`Identity Provider ` settings for the MinIO Tenant. -This includes configuring an external IDP such as :ref:`OpenID ` or :ref:`Active Directory / LDAP `. - -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% - - * - Field - - Description - - * - :guilabel:`Built-In` - - Configure additional internal MinIO users for the Operator to create as part of deploying the Tenant. - - * - :guilabel:`OpenID` - - Configure an OpenID Connect-compatible service as an external Identity Provider (e.g. Keycloak, Okta, Google, Facebook, Dex) to manage MinIO users. - - * - :guilabel:`Active Directory` - - Configure an Active Directory or OpenLDAP service as the external Identity Provider to manage MinIO users. - -.. _create-tenant-security-section: - -.. _minio-k8s-deploy-minio-tenant-security: - -7) The :guilabel:`Security` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The :guilabel:`Security` section displays TLS certificate settings for the MinIO Tenant. - -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% - - * - Field - - Description - - * - :guilabel:`TLS` - - Enable or disable TLS for the MinIO Tenant. - - * - :guilabel:`AutoCert` - - Directs the Operator to generate Certificate Signing Requests for submission to the Kubernetes TLS API. - - The MinIO Tenant uses the generated certificates for enabling and establishing TLS connections. - - * - :guilabel:`Custom Certificates` - - When enabled, you can upload custom TLS certificates for MinIO to use for server and client credentials. - - MinIO supports Server Name Indication (SNI) such that the Tenant can select the appropriate TLS certificate based on the request hostname and the certificate Subject Alternative Name. - - MinIO also supports uploading Certificate Authority certificates for validating client certificates minted by that CA. - -.. admonition:: Supported Secret Types - :class: note - - MinIO supports three types of :kube-docs:`secrets in Kubernetes `. - - #. ``opaque`` - - Using ``private.key`` and ``public.crt`` files. - #. ``tls`` - - Using ``tls.key`` and ``tls.crt`` files. - #. `cert-manager `__ 1.7.x or later - - Running on Kubernetes 1.21 or later. - -.. versionadded:: Console 0.23.1 - - A message displays under the certificate with the date of expiration and length of time until expiration. - - The message adjusts depending on the length of time to expiration: - - - More than 30 days, the message text displays in gray. - - Within 30 days, the message text changes to orange. - - Within 10 days, the message text changes to red. - - Within 24 hours, the message displays as an hour and minute countdown in red text. - - After expiration, the message displays as ``EXPIRED``. - -.. _create-tenant-encryption-section: - -8) The :guilabel:`Encryption` Section -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The :guilabel:`Encryption` section displays the :ref:`Server-Side Encryption (SSE) ` settings for the MinIO Tenant. - -Enabling SSE also creates :minio-git:`MinIO Key Encryption Service ` pods in the Tenant to facilitate SSE operations. - -.. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% - - * - Field - - Description - - * - :guilabel:`Vault` - - Configure `HashiCorp Vault `__ as the external KMS for storing root encryption keys. - See :ref:`minio-sse-vault` for guidance on the displayed fields. - - * - :guilabel:`AWS` - - Configure `AWS Secrets Manager `__ as the external KMS for storing root encryption keys. - See :ref:`minio-sse-aws` for guidance on the displayed fields. - - * - :guilabel:`Gemalto` - - Configure `Gemalto (Thales Digital Identity and Security) `__ as the external KMS for storing root encryption keys. - See :kes-docs:`Thales CipherTrust Manager (formerly Gemalto KeySecure) ` for guidance on the displayed fields. - - * - :guilabel:`GCP` - - Configure `Google Cloud Platform Secret Manager `__ as the external KMS for storing root encryption keys. - See :ref:`minio-sse-gcp` for guidance on the displayed fields. - - * - :guilabel:`Azure` - - Configure `Azure Key Vault `__ as the external KMS for storing root encryption keys. - See :ref:`minio-sse-azure` for guidance on the displayed fields. - -.. _minio-tenant-audit-logging-settings: - -.. _create-tenant-deploy-view-tenant: - -9) Deploy and View the Tenant -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Select :guilabel:`Create` at any time to begin the deployment process. -The MinIO Operator displays the root user credentials *once* as part of deploying the Tenant. -Copy these credentials to a secure location. - -You can monitor the Tenant creation process from the :guilabel:`Tenants` view. -The :guilabel:`State` column updates throughout the deployment process. - -Tenant deployment can take several minutes to complete. -Once the :guilabel:`State` reads as :guilabel:`Initialized`, select the Tenant to view its details. - -.. screenshot temporarily removed - - .. image:: /images/k8s/operator-tenant-view.png - :align: center - :width: 70% - :class: no-scaled-link - :alt: Tenant View - -Each tab provides additional details or configuration options for the MinIO Tenant. - -- :guilabel:`METRICS` - Displays metrics collected from the MinIO Tenant. -- :guilabel:`SECURITY` - Provides TLS-related configuration options. -- :guilabel:`POOLS` - Supports expanding the tenant by adding more Server Pools. -- :guilabel:`LICENSE` - Enter your `SUBNET `__ license. + If you deployed your MinIO Tenant using TLS certificates minted by a trusted Certificate Authority (CA) you can omit the ``--insecure`` flag. + + See :ref:`create-tenant-connect-tenant` for specific instructions. .. _create-tenant-connect-tenant: -10) Connect to the Tenant -~~~~~~~~~~~~~~~~~~~~~~~~~ +Connect to the Tenant +--------------------- The MinIO Operator creates services for the MinIO Tenant. @@ -671,7 +390,7 @@ The MinIO Operator creates services for the MinIO Tenant. .. code-block:: shell :class: copyable - oc get svc -n minio-tenant-1 + oc get svc -n TENANT-NAMESPACE .. cond:: k8s and not openshift @@ -680,17 +399,14 @@ The MinIO Operator creates services for the MinIO Tenant. .. code-block:: shell :class: copyable - kubectl get svc -n minio-tenant-1 + kubectl get svc -n TENANT-NAMESPACE .. code-block:: shell NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minio LoadBalancer 10.97.114.60 443:30979/TCP 2d3h - minio-tenant-1-console LoadBalancer 10.106.103.247 9443:32095/TCP 2d3h - minio-tenant-1-hl ClusterIP None 9000/TCP 2d3h - minio-tenant-1-log-hl-svc ClusterIP None 5432/TCP 2d3h - minio-tenant-1-log-search-api ClusterIP 10.103.5.235 8080/TCP 2d3h - minio-tenant-1-prometheus-hl-svc ClusterIP None 9090/TCP 7h39m + TENANT-NAMESPACE-console LoadBalancer 10.106.103.247 9443:32095/TCP 2d3h + TENANT-NAMESPACE-hl ClusterIP None 9000/TCP 2d3h - The ``minio`` service corresponds to the MinIO Tenant service. Applications should use this service for performing operations against the MinIO Tenant. diff --git a/source/operations/install-deploy-manage/deploy-operator-helm.rst b/source/operations/install-deploy-manage/deploy-operator-helm.rst index e0a6c304..6c2e952a 100644 --- a/source/operations/install-deploy-manage/deploy-operator-helm.rst +++ b/source/operations/install-deploy-manage/deploy-operator-helm.rst @@ -21,13 +21,11 @@ The following procedure uses a Helm Chart to install the :ref:`MinIO Kubernetes Prerequisites ------------- -To install the Operator with Helm you will need the following: +See the :ref:`Operator Prerequisites ` for a baseline of requirements. +Helm installations have the following additional requirements: -* An existing Kubernetes cluster. -* The ``kubectl`` CLI tool on your local host, the same version as the cluster. -* `Helm `__ version 3.8 or greater. -* `yq `__ version 4.18.1 or greater. -* Access to run ``kubectl`` commands on the cluster from your local host. +* `Helm `__ (Use the Version appropriate for your Kubernetes API version) +* `yq `__ For more about Operator installation requirements, including supported Kubernetes versions and TLS certificates, see the :ref:`Operator deployment prerequisites `. @@ -72,14 +70,16 @@ You can modify the Operator deployment after installation. NAME CHART VERSION APP VERSION DESCRIPTION minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator - minio-operator/operator 5.0.10 v5.0.10 A Helm chart for MinIO Operator - minio-operator/tenant 5.0.10 v5.0.10 A Helm chart for MinIO Operator + minio-operator/operator 6.0.1 v6.0.1 A Helm chart for MinIO Operator + minio-operator/tenant 6.0.1 v6.0.1 A Helm chart for MinIO Operator The ``minio-operator/minio-operator`` is a legacy chart and should **not** be installed under normal circumstances. #. Install the Operator - Run the ``helm install`` command to install the Operator: + Run the ``helm install`` command to install the Operator. + The following command specifies and creates a dedicated namespace ``minio-operator`` for installation. + MinIO strongly recommends using a dedicated namespace for the Operator. .. code-block:: shell :class: copyable @@ -103,86 +103,20 @@ You can modify the Operator deployment after installation. .. code-block:: shell NAME READY STATUS RESTARTS AGE - pod/console-68d955874d-vxlzm 1/1 Running 0 25h pod/minio-operator-699f797b8b-th5bk 1/1 Running 0 25h pod/minio-operator-699f797b8b-nkrn9 1/1 Running 0 25h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/console ClusterIP 10.43.195.224 9090/TCP,9443/TCP 25h service/operator ClusterIP 10.43.44.204 4221/TCP 25h service/sts ClusterIP 10.43.70.4 4223/TCP 25h NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/console 1/1 1 1 25h deployment.apps/minio-operator 2/2 2 2 25h NAME DESIRED CURRENT READY AGE - replicaset.apps/console-68d955874d 1 1 1 25h - replicaset.apps/minio-operator-699f797b8b 2 2 2 25h + replicaset.apps/minio-operator-79f7bfc48 2 2 2 123m -#. *(Optional)* Enable NodePort Access to the Console - - The Operator Console service does not automatically bind or expose itself for external access on the Kubernetes cluster. - You must instead configure a network control plane component, such as a load balancer or ingress, to grant that external access. - - You can enable :kube-docs:`Node Port ` access to the ``service/console`` service to allow simplified access to the MinIO Operator. - You should skip this step if you intend to configure the Operator Console service to use a Kubernetes Load Balancer, ingress, or similar control plane component that enables external access. - - Edit the ``service/console`` and set the ``spec.ports[0].nodePort`` and ``spec.type`` fields as follows: - - .. code-block:: yaml - - spec: - ports: - - name: http - port: 9090 - protocol: TCP - targetPort: 9090 - nodePort: 30090 - type: NodePort - - You can attempt to connect to the MinIO Operator Console by specifying port ``30090`` on any of the worker nodes in the deployment. - -#. Retrieve the Console Access Token - - The MinIO Operator uses a JSON Web Token (JWT) saved as a Kubernetes Secret for controlling access to the Operator Console. - - Use the following command to retrieve the JWT for login. - You must have permission within the Kubernetes cluster to read secrets: - - .. code-block:: shell - :class: copyable - - kubectl get secret/console-sa-secret -n minio-operator -o json | jq -r ".data.token" | base64 -d - - The output should resemble the following: - - .. code-block:: shell - - eyJhbGciOiJSUzI1NiIsImtpZCI6IlRtV2x3Z1RILVREaThhQm9iemFfLW95NHFHT0ZZOHFBRjlZalBRcWZiSDgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb25zb2xlLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiY2M1ZjEwYzktYzU1ZC00MjNiLTgxM2MtNmU5ZDY2ZGI5NDYyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1pbmlvLW9wZXJhdG9yOmNvbnNvbGUtc2EifQ.F-Pt5nU9xaugjRksWAOTShBW_eNTf8UwXvLfGxEK6l3_41NYsLgvTg5m0hYLUiYr6v2HwkEu0XzqTJbPoeSrFds8BOjeiCoP2Lmw4tRPo9tSXhAq-_elWt83YpJl-zjUpna5nVSWJWXKgj1Iga-9gw-Q63UygEcyTJ9_AwCNU9T0HdPzqccS9XrEUdsXFQxR9RwZY4TGC8K7cD9sc_OmfEiuyilRgyC_gFRvtCQfFv1DP0GKyjMGo2ffu-2Tq2U7zK5epWdqmNSvbIa0ZRoPlPedZ6nYY935lNgTIIW1oykRYrgwZZiv4CzfTH2gPswjtPc5ICtDDRUjYEhdTq3gtw - - If the output includes a trailing ``%`` make sure to omit it from the result. - -#. Log into the MinIO Operator Console - - If you configured the ``svc/console`` service for access through ingress, a cluster load balancer, you can access the Console using the configured hostname and port. - - If you configured the service for access through NodePorts, specify the hostname of any worker node in the cluster with that port as ``HOSTNAME:NODEPORT`` to access the Console. - - Alternatively, you can use ``kubectl port forward`` to temporary forward ports for the Console: - - .. code-block:: shell - :class: copyable - - kubectl port-forward svc/console -n minio-operator 9090:9090 - - You can then use ``http://localhost:9090`` to access the MinIO Operator Console. - - Once you access the Console, use the Console JWT to log in. - -You can now :ref:`deploy and manage MinIO Tenants using the Operator Console `. - -You can also :ref:`deploy a tenant using Helm Charts `. +You can now :ref:`deploy a tenant using Helm Charts `. .. _minio-k8s-deploy-operator-helm-local: @@ -202,224 +136,29 @@ This method may support easier pre-configuration of the Operator compared to the curl -O https://raw.githubusercontent.com/minio/operator/master/helm-releases/operator-|operator-version-stable|.tgz + +#. (Optional) Modify the ``values.yaml`` + The chart contains a ``values.yaml`` file you can customize to suit your needs. For details on the options available in the MinIO Operator ``values.yaml``, see :ref:`minio-operator-chart-values`. + For example, you can change the number of replicas for ``operators.replicaCount`` to increase or decrease pod availability in the deployment. + See :ref:`minio-operator-chart-values` for more complete documentation on the Operator Helm Chart and Values. + For more about customizations, see `Helm Charts `__. - -#. Deploy Operator - The following Helm command deploys the MinIO Operator using the downloaded chart: +#. Install the Helm Chart + + Use the ``helm install`` command to install the chart. + The following command assumes the Operator chart is saved to ``./operator`` relative to the working directory. .. code-block:: shell :class: copyable - :substitutions: helm install \ --namespace minio-operator \ --create-namespace \ - operator operator-|operator-version-stable|.tgz - -#. Configure Operator - - A. Create the YAML configuration files - - Use ``kubectl`` and ``yq`` to create the following files: - - * service.yaml: - - .. code-block:: shell - :class: copyable - - kubectl get service console -n minio-operator -o yaml > service.yaml - yq e -i '.spec.type="NodePort"' service.yaml - yq e -i '.spec.ports[0].nodePort = PORT_NUMBER' service.yaml - - Replace ``PORT_NUMBER`` with the port on which to serve the Operator GUI. - - The file contents resemble the following: - - .. dropdown:: Example ``service.yaml`` file - - .. code-block:: yaml - - apiVersion: v1 - kind: Service - metadata: - annotations: - meta.helm.sh/release-name: minio-operator - meta.helm.sh/release-namespace: minio-operator - creationTimestamp: "2023-05-11T14:57:42Z" - labels: - app.kubernetes.io/instance: minio-operator - app.kubernetes.io/managed-by: Helm - app.kubernetes.io/name: operator - app.kubernetes.io/version: v5.0.4 - helm.sh/chart: operator-5.0.4 - name: console - namespace: minio-operator - resourceVersion: "907" - uid: 9297fd97-806a-4715-8bd5-a1f6103149a8 - spec: - clusterIP: 10.96.157.135 - clusterIPs: - - 10.96.157.135 - internalTrafficPolicy: Cluster - ipFamilies: - - IPv4 - ipFamilyPolicy: SingleStack - ports: - - name: http - port: 9090 - protocol: TCP - targetPort: 9090 - nodePort: 30080 - - name: https - port: 9443 - protocol: TCP - targetPort: 9443 - selector: - app.kubernetes.io/instance: minio-operator-console - app.kubernetes.io/name: operator - sessionAffinity: None - type: NodePort - status: - loadBalancer: {} - - * operator.yaml: - - .. code-block:: shell - :class: copyable - - kubectl get deployment minio-operator -n minio-operator -o yaml > operator.yaml - yq -i -e '.spec.replicas |= 1' operator.yaml - - The file contents resemble the following: - - .. dropdown:: Example ``operator.yaml`` file - - .. code-block:: shell - - apiVersion: apps/v1 - kind: Deployment - metadata: - annotations: - deployment.kubernetes.io/revision: "1" - meta.helm.sh/release-name: minio-operator - meta.helm.sh/release-namespace: minio-operator - creationTimestamp: "2023-05-11T14:57:43Z" - generation: 1 - labels: - app.kubernetes.io/instance: minio-operator - app.kubernetes.io/managed-by: Helm - app.kubernetes.io/name: operator - app.kubernetes.io/version: v5.0.4 - helm.sh/chart: operator-5.0.4 - name: minio-operator - namespace: minio-operator - resourceVersion: "947" - uid: f395171e-d17c-4645-9854-3dd92f23be59 - spec: - progressDeadlineSeconds: 600 - replicas: 1 - revisionHistoryLimit: 10 - selector: - matchLabels: - app.kubernetes.io/instance: minio-operator - app.kubernetes.io/name: operator - strategy: - rollingUpdate: - maxSurge: 25% - maxUnavailable: 25% - type: RollingUpdate - template: - metadata: - creationTimestamp: null - labels: - app.kubernetes.io/instance: minio-operator - app.kubernetes.io/name: operator - spec: - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: name - operator: In - values: - - minio-operator - topologyKey: kubernetes.io/hostname - containers: - - args: - - controller - image: quay.io/minio/operator:v5.0.4 - imagePullPolicy: IfNotPresent - name: operator - resources: - requests: - cpu: 200m - ephemeral-storage: 500Mi - memory: 256Mi - securityContext: - runAsGroup: 1000 - runAsNonRoot: true - runAsUser: 1000 - terminationMessagePath: /dev/termination-log - terminationMessagePolicy: File - dnsPolicy: ClusterFirst - restartPolicy: Always - schedulerName: default-scheduler - securityContext: - fsGroup: 1000 - runAsGroup: 1000 - runAsNonRoot: true - runAsUser: 1000 - serviceAccount: minio-operator - serviceAccountName: minio-operator - terminationGracePeriodSeconds: 30 - status: - conditions: - - lastTransitionTime: "2023-05-11T14:57:43Z" - lastUpdateTime: "2023-05-11T14:57:43Z" - message: Deployment does not have minimum availability. - reason: MinimumReplicasUnavailable - status: "False" - type: Available - - lastTransitionTime: "2023-05-11T14:57:43Z" - lastUpdateTime: "2023-05-11T14:57:44Z" - message: ReplicaSet "minio-operator-674cf5cf78" is progressing. - reason: ReplicaSetUpdated - status: "True" - type: Progressing - observedGeneration: 1 - replicas: 2 - unavailableReplicas: 2 - updatedReplicas: 2 - - * console-secret.yaml: - - Create a ``console-secret.yaml`` file with the following contents: - - .. code-block:: shell - :class: copyable - - apiVersion: v1 - kind: Secret - metadata: - name: console-sa-secret - namespace: minio-operator - annotations: - kubernetes.io/service-account.name: console-sa - type: kubernetes.io/service-account-token - - B. Apply the configuration to your deployment with ``kubectl apply``: - - .. code-block:: shell - :class: copyable - - kubectl apply -f service.yaml - kubectl apply -f operator.yaml - kubectl apply -f console-secret.yaml + minio-operator ./operator #. To verify the installation, run the following command: @@ -436,94 +175,16 @@ This method may support easier pre-configuration of the Operator compared to the .. code-block:: shell NAME READY STATUS RESTARTS AGE - pod/console-59b769c486-cv7zv 1/1 Running 0 81m pod/minio-operator-7976b4df5b-rsskl 1/1 Running 0 81m + pod/minio-operator-7976b4df5b-x622g 1/1 Running 0 81m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE - service/console ClusterIP 10.105.218.94 9090/TCP,9443/TCP 81m service/operator ClusterIP 10.110.113.146 4222/TCP,4233/TCP 81m NAME READY UP-TO-DATE AVAILABLE AGE - deployment.apps/console 1/1 1 1 81m - deployment.apps/minio-operator 1/1 1 1 81m + deployment.apps/minio-operator 2/2 2 2 81m NAME DESIRED CURRENT READY AGE - replicaset.apps/console-59b769c486 1 1 1 81m replicaset.apps/minio-operator-7976b4df5b 1 1 1 81m - -#. Connect to the Operator Console - - To connect to the Console, first retrieve the JSON Web Token (JWT) for your deployment and then forward the Console port. - - A. Retrieve the JWT - - The Operator Console uses a JWT to authenticate and log in. - The following commands retrieve the token for your deployment: - - .. code-block:: shell - :class: copyable - - SA_TOKEN=$(kubectl -n minio-operator get secret console-sa-secret -o jsonpath="{.data.token}" | base64 --decode) - echo $SA_TOKEN - - B. Forward the Operator Console port to allow access from another host. - - The following command temporarily forwards the Console to port 9090: - - .. code-block:: shell - :class: copyable - - kubectl --namespace minio-operator port-forward svc/console 9001:9001 - - This command forwards the pod port ``9090`` to the matching port on the local machine while active in the shell. - The ``kubectl port-forward`` command only functions while active in the shell session. - Terminating the session closes the ports on the local machine. - - C. Access the Console by navigating to ``http://localhost:9001`` in a browser and login with the JWT. - - .. note:: - - Some Kubernetes deployments may experience issues with timeouts during port-forwarding operations with the Operator Console. - Select the :guilabel:`NodePorts` section to view instructions for alternative access. - You can alternatively configure your preferred Ingress to grant access to the Operator Console service. - See https://github.com/kubernetes/kubectl/issues/1368 for more information. - -.. dropdown:: NodePorts - - Use the following command to identify the :kube-docs:`NodePorts ` configured for the Operator Console. - If your local host does not have the ``jq`` utility installed, you can run the first command and locate the ``spec.ports`` section of the output. - - .. code-block:: shell - :class: copyable - - kubectl get svc/console -n minio-operator -o json | jq -r '.spec.ports' - - The output resembles the following: - - .. code-block:: json - - [ - { - "name": "http", - "nodePort": 31055, - "port": 9090, - "protocol": "TCP", - "targetPort": 9090 - }, - { - "name": "https", - "nodePort": 31388, - "port": 9443, - "protocol": "TCP", - "targetPort": 9443 - } - ] - - Use the ``http`` or ``https`` port depending on whether you deployed the Operator with Console TLS enabled via :mc-cmd:`kubectl minio init --console-tls`. - - Append the ``nodePort`` value to the externally-accessible IP address of a worker node in your Kubernetes cluster. - -You can now :ref:`deploy and manage MinIO Tenants using the Operator Console `. - -You can also :ref:`deploy a tenant using Helm Charts `. +You can now :ref:`deploy a tenant using Helm Charts `. diff --git a/source/operations/install-deploy-manage/expand-minio-tenant.rst b/source/operations/install-deploy-manage/expand-minio-tenant.rst index 48303d07..c56fecaf 100644 --- a/source/operations/install-deploy-manage/expand-minio-tenant.rst +++ b/source/operations/install-deploy-manage/expand-minio-tenant.rst @@ -13,6 +13,12 @@ Expand a MinIO Tenant This procedure documents expanding the available storage capacity of an existing MinIO tenant by deploying an additional pool of MinIO pods in the Kubernetes infrastructure. +.. important:: + + The MinIO Operator Console is deprecated and removed in Operator 6.0.0. + + See :ref:`minio-k8s-modify-minio-tenant` for instructions on migrating Tenants installed via the Operator Console to Kustomization. + Prerequisites ------------- @@ -55,44 +61,83 @@ Persistent Volumes MinIO strongly recommends using SSD-backed EBS volumes for best performance. For more information on EBS resources, see `EBS Volume Types `__. -Procedure (Operator Console) ----------------------------- +Procedure +--------- + +The MinIO Operator supports expanding a MinIO Tenant by adding additional pools. + +.. tab-set:: + + .. tab-item:: Kustomization + + #. Review the Kustomization object which describes the Tenant object (``tenant.yaml``). + + The ``spec.pools`` array describes the current pool topology. + + #. Add a new entry to the ``spec.pools`` array. + + The new pool must reflect your intended combination of Worker nodes, volumes per server, storage class, and affinity/scheduler settings. + See :ref:`minio-operator-crd` for more complete documentation on Pool-related configuration settings. + + #. Apply the updated Tenant configuration + + Use the ``kubectl apply`` command to update the Tenant: + + .. code-block:: shell + + kubectl apply -k ~/kustomization/TENANT-NAME + + Modify the path to the Kustomization directory to match your local configuration. + + .. tab-item:: Helm + + #. Review the Helm ``values.yaml`` file. + + The ``tenant.pools`` array describes the current pool topology. + + #. Add a new entry to the ``tenant.pools`` array. + + The new pool must reflect your intended combination of Worker nodes, volumes per server, storage class, and affinity/scheduler settings. + See :ref:`minio-tenant-chart-values` for more complete documentation on Pool-related configuration settings. + + #. Apply the updated Tenant configuration + + Use the ``helm upgrade`` command to update the Tenant: + + .. code-block:: shell + + helm upgrade TENANT-NAME minio-operator/tenant -f values.yaml -n TENANT-NAMESPACE + + The command above assumes use of the MinIO Operator Chart repository. + If you installed the Chart manually or by using a different repository name, specify that chart or name in the command. -The MinIO Operator Console supports expanding a MinIO Tenant by adding additional pools. + Replace ``TENANT-NAME`` and ``TENANT-NAMESPACE`` with the name and namespace of the Tenant respectively. + You can use ``helm list -n TENANT-NAMESPACE`` to validate the Tenant name. +You can use the ``kubectl get events -n TENANT-NAMESPACE --watch`` to monitor the progress of expansion. +The MinIO Operator updates services to route connections appropriately across the new nodes. +If you use customized services, routes, ingress, or similar Kubernetes network components, you may need to update those components for the new pod hostname ranges. -1) Expand the MinIO Tenant -~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. Following link is intended for K8s only +.. _minio-decommissioning: -#. From the Operator Console view, select the Tenant to open the summary view, then select :guilabel:`Pools`. - Select :guilabel:`Expand Tenant`. +Decommission a Tenant Server Pool +---------------------------------- -#. Specify the following information for the new pool: +Decommissioning a server pool involves three steps: - .. list-table:: - :header-rows: 1 - :widths: 30 70 - :width: 100% +1) Run the :mc-cmd:`mc admin decommission start` command against the Tenant - * - Field - - Description +2) Wait until decommissioning completes - * - Number of Servers - - The number of servers to deploy in the new Tenant Pool across the Kubernetes cluster. - - * - Volume Size - - The capacity of each volume in the new Tenant Pool. - - * - Volumes per Server - - The number of volumes for each server in the new Tenant Pool. +3) Modify the Tenant YAML to remove the decommissioned pool - * - Storage Class - - Specify the Kubernetes Storage Class the Operator uses when generating Persistent Volume Claims for the Tenant. - -#. Select :guilabel:`Create`. +When removing the Tenant pool, ensure the ``spec.pools.[n].name`` fields have values for all remaining pools. +.. include:: /includes/common-installation.rst + :start-after: start-pool-order-must-not-change + :end-before: end-pool-order-must-not-change -2) Validate the Expanded MinIO Tenant -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. important:: -In the :guilabel:`Pools` tab, select the new Pool to confirm its details. + You cannot reuse the same pool name or hostname sequence for a decommissioned pool. \ No newline at end of file diff --git a/source/operations/install-deploy-manage/minio-operator-console.rst b/source/operations/install-deploy-manage/minio-operator-console.rst index f6fd886f..31886568 100644 --- a/source/operations/install-deploy-manage/minio-operator-console.rst +++ b/source/operations/install-deploy-manage/minio-operator-console.rst @@ -1,3 +1,5 @@ +:orphan: + .. _minio-operator-console: ====================== @@ -10,6 +12,14 @@ MinIO Operator Console :local: :depth: 2 +.. warning:: + + MinIO Operator 6.0.0 deprecates and removes the Operator Console. + + You can use either Kustomization or Helm to manage and deploy MinIO Tenants. + + This page provides a historical view at the Operator Console, and will recieve no further updates or corrections. + The Operator Console provides a rich user interface for deploying and managing MinIO Tenants on Kubernetes infrastructure. Installing the MinIO :ref:`Kubernetes Operator ` automatically diff --git a/source/operations/install-deploy-manage/modify-minio-tenant.rst b/source/operations/install-deploy-manage/modify-minio-tenant.rst index 41b83289..67e73b05 100644 --- a/source/operations/install-deploy-manage/modify-minio-tenant.rst +++ b/source/operations/install-deploy-manage/modify-minio-tenant.rst @@ -10,87 +10,37 @@ Modify a MinIO Tenant :local: :depth: 1 -The procedures on this page use the :ref:`MinIO Operator Console ` for modifying an existing tenant. +You can modify tenants after deployment to change mutable configuration settings. +See :ref:`minio-operator-crd` for a complete description of available settings in the MinIO Custom Resource Definition. -.. screenshot temporarily removed - .. image:: /images/k8s/operator-manage-tenant.png - :align: center - :width: 70% - :class: no-scaled-link - :alt: MinIO Operator Tenant Console +The method for modifying the Tenant depends on how you deployed the tenant: -Certificate Management ----------------------- +.. tab-set:: -The Security section provides tools for adding and managing certificates for the tenant. + .. tab-item:: Kustomize + :sync: kustomize -Review Certificate expiration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + For Kustomize-deployed Tenants, you can modify the base Kustomization resources and apply them using ``kubectl apply -k`` against the directory containing the ``kustomization.yaml`` object. -.. versionadded:: Console 0.23.1 + .. code-block:: shell -A message displays under the certificate with the date of expiration and length of time until expiration. + kubectl apply -k ~/kustomization/TENANT-NAME/ -The message adjusts depending on the length of time to expiration: - -- More than 30 days, the message text displays in gray. -- Within 30 days, the message text changes to orange. -- Within 10 days, the message text changes to red. -- Within 24 hours, the message displays as an hour and minute countdown in red text. -- After expiration, the message displays as ``EXPIRED``. + Modify the path to the Kustomization directory to match your local configuration. -.. _minio-k8s-modify-minio-tenant-security: + .. tab-item:: Helm + :sync: helm -Modify Tenant TLS Configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + For Helm-deployed Tenants, you can modify the base ``values.yaml`` and upgrade the Tenant using the chart: -The MinIO Operator Console supports adding and removing TLS certificates from a MinIO Tenant. + .. code-block:: shell -From the Operator Console view, select the Tenant to open the summary view, then select :guilabel:`Security`. -You can make the following modifications: + helm upgrade TENANT-NAME minio-operator/tenant -f values.yaml -n TENANT-NAMESPACE -Enable or Disable TLS - Toggle the :guilabel:`TLS` switch to direct the Operator to either enable or disable TLS for the deployment. - The MinIO Operator automatically generates the necessary TLS certificates using the Kubernetes TLS API. - See :ref:`minio-tls-user-generated` for more information. + The command above assumes use of the MinIO Operator Chart repository. + If you installed the Chart manually or by using a different repository name, specify that chart or name in the command. -Add Custom TLS Certificates - MinIO Tenants support `Server Name Indication (SNI) `__, where the MinIO server identifies which certificate to use based on the hostname specified by the connecting client. - The MinIO Operator can attach additional TLS certificates to the Tenant to enable SNI-based TLS connectivity. + Replace ``TENANT-NAME`` and ``TENANT-NAMESPACE`` with the name and namespace of the Tenant, respectively. + You can use ``helm list -n TENANT-NAMESPACE`` to validate the Tenant name. - To customize the TLS certificates mounted on the MinIO Tenant, enable the :guilabel:`Custom Certificates` switch. - Select the :guilabel:`Add Certificate +` button to add custom TLS certificates. - -Add Trusted Certificate Authorities - The MinIO Tenant validates the TLS certificate presented by each connecting client against the host system's trusted root certificate store. - The MinIO Operator can attach additional third-party Certificate Authorities (CA) to the Tenant to allow validation of client TLS certificates signed by those CAs. - - To customize the trusted CAs mounted to each Tenant MinIO pod, enable the :guilabel:`Custom Certificates` switch. - Select the :guilabel:`Add CA Certificate +` button to add third party CA certificates. - - If the MinIO Tenant cannot match an incoming client's TLS certificate issuer against either the container OS's trust store *or* an explicitly attached CA, MinIO rejects the connection as invalid. - - -Manage Tenant Pools -------------------- - -Specify Runtime Class -~~~~~~~~~~~~~~~~~~~~~ - -.. versionadded:: Console 0.23.1 - -When adding a new pool or modifying an existing pool for a tenant, you can specify the :kube-docs:`Runtime Class Name ` for pools to use. - -.. Following link is intended for K8s only -.. _minio-decommissioning: - -Decommission a Tenant Server Pool -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MinIO Operator 4.4.13 and later support decommissioning a server pool in a Tenant. -Specifically, you can follow the :minio-docs:`Decommission a Server pool ` procedure to remove the pool from the tenant, then edit the tenant YAML to drop the pool from the StatefulSet. -When removing the Tenant pool, ensure the ``spec.pools.[n].name`` fields have values for all remaining pools. - -.. include:: /includes/common-installation.rst - :start-after: start-pool-order-must-not-change - :end-before: end-pool-order-must-not-change + See :ref:`minio-tenant-chart-values` for more complete documentation on the available Chart fields. \ No newline at end of file diff --git a/source/operations/install-deploy-manage/upgrade-minio-operator-4.5.7-earlier.rst b/source/operations/install-deploy-manage/upgrade-minio-operator-4.5.7-earlier.rst index 0cf7eaea..d4b38fb2 100644 --- a/source/operations/install-deploy-manage/upgrade-minio-operator-4.5.7-earlier.rst +++ b/source/operations/install-deploy-manage/upgrade-minio-operator-4.5.7-earlier.rst @@ -1,9 +1,7 @@ :orphan: -.. _minio-k8s-upgrade-minio-operator-to-4.5.8: - ================================ -Upgrade MinIO Operator to v4.5.8 +Upgrade Legacy MinIO Operators ================================ .. default-domain:: minio @@ -13,10 +11,7 @@ Upgrade MinIO Operator to v4.5.8 :depth: 1 -To upgrade from Operator to |operator-version-stable| from version 4.5.7 or earlier, you must first upgrade to version 4.5.8. -Depending on your current version, you may need to do one or more intermediate upgrades to reach v4.5.8. - -The following table lists the upgrade paths for older versions of MinIO Operator: +MinIO supports the following upgrade paths for older versions of the MinIO Operator: .. list-table:: :header-rows: 1 @@ -35,6 +30,288 @@ The following table lists the upgrade paths for older versions of MinIO Operator * - 3.X.X - 4.2.2 +To upgrade from Operator to |operator-version-stable| from version 4.5.7 or earlier, you must first upgrade to version 4.5.8. +Depending on your current version, you may need to do one or more intermediate upgrades to reach v4.5.8. + +.. _minio-k8s-upgrade-minio-operator-to-5.0.15: + +Upgrade MinIO Operator 4.5.8 and Later to 5.0.15 +------------------------------------------------ + +.. admonition:: Prerequisites + :class: note + + This procedure requires the following: + + - You have an existing MinIO Operator deployment running 4.5.8 or later + - Your Kubernetes cluster runs 1.21.0 or later + - Your local host has ``kubectl`` installed and configured with access to the Kubernetes cluster + +This procedure upgrades the MinIO Operator from any 4.5.8 or later release to 5.0.15 + +Tenant Custom Resource Definition Changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following changes apply for Operator v5.0.0 or later: + +- The ``.spec.s3`` field is replaced by the ``.spec.features`` field. +- The ``.spec.credsSecret`` field is replaced by the ``.spec.configuration`` field. + + The ``.spec.credsSecret`` should hold all the environment variables for the MinIO deployment that contain sensitive information and should not show in ``.spec.env``. + This change impacts the Tenant :abbr:`CRD (CustomResourceDefinition)` and only impacts users editing a tenant YAML directly, such as through Helm or Kustomize. +- Both the **Log Search API** (``.spec.log``) and **Prometheus** (``.spec.prometheus``) deployments have been removed. + However, existing deployments are left running as standalone deployments / statefulsets with no connection to the Tenant CR. + Deleting the Tenant :abbr:`CRD (Custom Resource Definition)` does **not** cascade to the log or Prometheus deployments. + + .. important:: + + MinIO recommends that you create a yaml file to manage these deployments going forward. + +Log Search and Prometheus +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The latest releases of Operator remove Log Search and Prometheus from included Operator tools. +The following steps back up the existing yaml files, perform some clean up, and provide steps to continue using either or both of these functions. + +#. Back up Prometheus and Log Search yaml files. + + .. code-block:: shell + :class: copyable + + export TENANT_NAME=myminio + export NAMESPACE=mynamespace + kubectl -n $NAMESPACE get secret $TENANT_NAME-log-secret -o yaml > $TENANT_NAME-log-secret.yaml + kubectl -n $NAMESPACE get cm $TENANT_NAME-prometheus-config-map -o yaml > $TENANT_NAME-prometheus-config-map.yaml + kubectl -n $NAMESPACE get sts $TENANT_NAME-prometheus -o yaml > $TENANT_NAME-prometheus.yaml + kubectl -n $NAMESPACE get sts $TENANT_NAME-log -o yaml > $TENANT_NAME-log.yaml + kubectl -n $NAMESPACE get deployment $TENANT_NAME-log-search-api -o yaml > $TENANT_NAME-log-search-api.yaml + kubectl -n $NAMESPACE get svc $TENANT_NAME-log-hl-svc -o yaml > $TENANT_NAME-log-hl-svc.yaml + kubectl -n $NAMESPACE get svc $TENANT_NAME-log-search-api -o yaml > $TENANT_NAME-log-search-api-svc.yaml + kubectl -n $NAMESPACE get svc $TENANT_NAME-prometheus-hl-svc -o yaml > $TENANT_NAME-prometheus-hl-svc.yaml + + - Replace ``myminio`` with the name of the tenant on the operator deployment you are upgrading. + - Replace ``mynamespace`` with the namespace for the tenant on the operator deployment you are upgrading. + + Repeat for each tenant. + +#. Remove ``.metadata.ownerReferences`` for all backed up files for all tenants. + +#. *(Optional)* To continue using Log Search API and Prometheus, add the following variables to the tenant's yaml specification file under ``.spec.env`` + + Use the following command to edit a tenant: + + .. code-block:: shell + :class: copyable + + kubectl edit tenants -n + + - Replace ```` with the name of the tenant to modify. + - Replace ```` with the namespace of the tenant you are modifying. + + Add the following values under ``.spec.env`` in the file: + + .. code-block:: yaml + :class: copyable + + - name: MINIO_LOG_QUERY_AUTH_TOKEN + valueFrom: + secretKeyRef: + key: MINIO_LOG_QUERY_AUTH_TOKEN + name: -log-secret + - name: MINIO_LOG_QUERY_URL + value: http://-log-search-api:8080 + - name: MINIO_PROMETHEUS_JOB_ID + value: minio-job + - name: MINIO_PROMETHEUS_URL + value: http://-prometheus-hl-svc:9001 + + - Replace ```` in the ``name`` or ``value`` lines with the name of your tenant. + +Procedure +~~~~~~~~~ + +.. tab-set:: + + .. tab-item:: Upgrade using Kustomize + + The following procedure upgrades the MinIO Operator using Kustomize. + + For Operator versions 5.0.1 to 5.0.14 installed with the MinIO Kubernetes Plugin, follow the Kustomize instructions to upgrade to 5.0.15 or later. + If you installed the Operator using :ref:`Helm `, use the :guilabel:`Upgrade using Helm` instructions instead. + + #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. + + Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. + Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. + See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. + + #. Verify the existing Operator installation. + Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. + + If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. + + You can verify the currently installed Operator version by retrieving the object specification for an operator pod in the namespace. + The following example uses the ``jq`` tool to filter the necessary information from ``kubectl``: + + .. code-block:: shell + :class: copyable + + kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' + + The output resembles the following: + + .. code-block:: json + :emphasize-lines: 8-10 + :substitutions: + + { + "env": [ + { + "name": "CLUSTER_DOMAIN", + "value": "cluster.local" + } + ], + "image": "minio/operator:v|operator-version-stable|", + "imagePullPolicy": "IfNotPresent", + "name": "minio-operator" + } + + If your local host does not have the ``jq`` utility installed, you can run the first part of the command and locate the ``spec.containers`` section of the output. + + #. Upgrade Operator with Kustomize + + The following command upgrades Operator to version |operator-version-stable|: + + .. code-block:: shell + :class: copyable + + kubectl apply -k github.com/minio/operator + + In the sample output below, ``configured`` at the end of the line indicates where a new change was applied from the updated CRD: + + .. code-block:: shell + + namespace/minio-operator configured + customresourcedefinition.apiextensions.k8s.io/miniojobs.job.min.io configured + customresourcedefinition.apiextensions.k8s.io/policybindings.sts.min.io configured + customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io configured + serviceaccount/console-sa unchanged + serviceaccount/minio-operator unchanged + clusterrole.rbac.authorization.k8s.io/console-sa-role unchanged + clusterrole.rbac.authorization.k8s.io/minio-operator-role unchanged + clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding unchanged + clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding unchanged + configmap/console-env unchanged + secret/console-sa-secret configured + service/console unchanged + service/operator unchanged + service/sts unchanged + deployment.apps/console configured + deployment.apps/minio-operator configured + + + #. Validate the Operator upgrade + + You can check the new Operator version with the same ``kubectl`` command used previously: + + .. code-block:: shell + :class: copyable + + kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' + + #. *(Optional)* Connect to the Operator Console + + .. include:: /includes/common/common-k8s-connect-operator-console-no-plugin.rst + + #. Retrieve the Operator Console JWT for login + + .. include:: /includes/common/common-k8s-operator-console-jwt.rst + + + .. tab-item:: Upgrade using Helm + + The following procedure upgrades an existing MinIO Operator Installation using Helm. + + If you installed the Operator using Kustomize, use the :guilabel:`Upgrade using Kustomize` instructions instead. + + #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. + + Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. + Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. + See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. + + #. Verify the existing Operator installation. + + Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. + + If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. + + Use the ``helm list`` command to view the installed charts in the namespace: + + .. code-block:: shell + :class: copyable + + helm list -n minio-operator + + The result should resemble the following: + + .. code-block:: shell + :class: copyable + + NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION + operator minio-operator 1 2023-11-01 15:49:54.539724775 -0400 EDT deployed operator-5.0.x v5.0.x + + #. Update the Operator Repository + + Use ``helm repo update minio-operator`` to update the MinIO Operator repo. + If you set a different alias for the MinIO Operator repository, specify that in the command instead of ``minio-operator``. + You can use ``helm repo list`` to review your installed repositories. + + Use ``helm search`` to check the latest available chart version after updating the Operator Repo: + + .. code-block:: shell + :class: copyable + + helm search repo minio-operator + + The response should resemble the following: + + .. code-block:: shell + :class: copyable + :substitutions: + + NAME CHART VERSION APP VERSION DESCRIPTION + minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator + minio-operator/operator |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator + minio-operator/tenant |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator + + The ``minio-operator/minio-operator`` is a legacy chart and should **not** be installed under normal circumstances. + + #. Run ``helm upgrade`` + + Helm uses the latest chart to upgrade the MinIO Operator: + + .. code-block:: shell + :class: copyable + + helm upgrade -n minio-operator \ + operator minio-operator/operator + + If you installed the MinIO Operator to a different namespace, specify that in the ``-n`` argument. + + If you used a different installation name from ``operator``, replace the value above with the installation name. + + The command results should return success with a bump in the ``REVISION`` value. + + #. Validate the Operator upgrade + + .. include:: /includes/common/common-k8s-connect-operator-console-no-plugin.rst + + #. Retrieve the Operator Console JWT for login + + .. include:: /includes/common/common-k8s-operator-console-jwt.rst + +.. _minio-k8s-upgrade-minio-operator-to-4.5.8: Upgrade MinIO Operator 4.2.3 through 4.5.7 to 4.5.8 --------------------------------------------------- diff --git a/source/operations/install-deploy-manage/upgrade-minio-operator.rst b/source/operations/install-deploy-manage/upgrade-minio-operator.rst index 9aea52a1..374e496d 100644 --- a/source/operations/install-deploy-manage/upgrade-minio-operator.rst +++ b/source/operations/install-deploy-manage/upgrade-minio-operator.rst @@ -15,203 +15,112 @@ You can upgrade the MinIO Operator at any time without impacting your managed Mi As part of the upgrade process, the Operator may update and restart Tenants to support changes to the MinIO Custom Resource Definition (CRD). These changes require no action on the part of any operator or administrator, and do not impact Tenant operations. -This page describes how to upgrade from Operator 4.5.8 or later to |operator-version-stable|. -To upgrade from Operator 4.5.7 or earlier, see :ref:`Upgrade MinIO Operator to v4.5.8 `. +This page describes how to upgrade from Operator 5.0.15 to |operator-version-stable|. +See :ref:`minio-k8s-upgrade-minio-operator-to-5.0.15` for instructions on upgrading to Operator 5.0.15 before starting this procedure. -.. _minio-k8s-upgrade-minio-operator-procedure: - -Upgrade MinIO Operator 4.5.8 and Later to |operator-version-stable| -------------------------------------------------------------------- - -.. admonition:: Prerequisites - :class: note - - This procedure requires the following: - - - You have an existing MinIO Operator deployment running 4.5.8 or later - - Your Kubernetes cluster runs 1.21.0 or later - - Your local host has ``kubectl`` installed and configured with access to the Kubernetes cluster - -This procedure upgrades the MinIO Operator from any 4.5.8 or later release to |operator-version-stable|. - -Tenant Custom Resource Definition Changes -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following changes apply for Operator v5.0.0 or later: - -- The ``.spec.s3`` field is replaced by the ``.spec.features`` field. -- The ``.spec.credsSecret`` field is replaced by the ``.spec.configuration`` field. - - The ``.spec.credsSecret`` should hold all the environment variables for the MinIO deployment that contain sensitive information and should not show in ``.spec.env``. - This change impacts the Tenant :abbr:`CRD (CustomResourceDefinition)` and only impacts users editing a tenant YAML directly, such as through Helm or Kustomize. -- Both the **Log Search API** (``.spec.log``) and **Prometheus** (``.spec.prometheus``) deployments have been removed. - However, existing deployments are left running as standalone deployments / statefulsets with no connection to the Tenant CR. - Deleting the Tenant :abbr:`CRD (Custom Resource Definition)` does **not** cascade to the log or Prometheus deployments. - - .. important:: - - MinIO recommends that you create a yaml file to manage these deployments going forward. - -Log Search and Prometheus -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The latest releases of Operator remove Log Search and Prometheus from included Operator tools. -The following steps back up the existing yaml files, perform some clean up, and provide steps to continue using either or both of these functions. -#. Back up Prometheus and Log Search yaml files. +.. admonition:: Operator 6.0.0 Deprecates the Operator Console - .. code-block:: shell - :class: copyable + Starting with Operator 6.0.0, the MinIO Operator Console is deprecated and removed. - export TENANT_NAME=myminio - export NAMESPACE=mynamespace - kubectl -n $NAMESPACE get secret $TENANT_NAME-log-secret -o yaml > $TENANT_NAME-log-secret.yaml - kubectl -n $NAMESPACE get cm $TENANT_NAME-prometheus-config-map -o yaml > $TENANT_NAME-prometheus-config-map.yaml - kubectl -n $NAMESPACE get sts $TENANT_NAME-prometheus -o yaml > $TENANT_NAME-prometheus.yaml - kubectl -n $NAMESPACE get sts $TENANT_NAME-log -o yaml > $TENANT_NAME-log.yaml - kubectl -n $NAMESPACE get deployment $TENANT_NAME-log-search-api -o yaml > $TENANT_NAME-log-search-api.yaml - kubectl -n $NAMESPACE get svc $TENANT_NAME-log-hl-svc -o yaml > $TENANT_NAME-log-hl-svc.yaml - kubectl -n $NAMESPACE get svc $TENANT_NAME-log-search-api -o yaml > $TENANT_NAME-log-search-api-svc.yaml - kubectl -n $NAMESPACE get svc $TENANT_NAME-prometheus-hl-svc -o yaml > $TENANT_NAME-prometheus-hl-svc.yaml + You can continue to manage and deploy MinIO Tenants using standard Kubernetes approaches such as Kustomize or Helm. - - Replace ``myminio`` with the name of the tenant on the operator deployment you are upgrading. - - Replace ``mynamespace`` with the namespace for the tenant on the operator deployment you are upgrading. - - Repeat for each tenant. - -#. Remove ``.metadata.ownerReferences`` for all backed up files for all tenants. - -#. *(Optional)* To continue using Log Search API and Prometheus, add the following variables to the tenant's yaml specification file under ``.spec.env`` - - Use the following command to edit a tenant: - - .. code-block:: shell - :class: copyable - - kubectl edit tenants -n - - - Replace ```` with the name of the tenant to modify. - - Replace ```` with the namespace of the tenant you are modifying. +.. _minio-k8s-upgrade-minio-operator-procedure: - Add the following values under ``.spec.env`` in the file: +Upgrade MinIO Operator 5.0.15 to |operator-version-stable| +---------------------------------------------------------- - .. code-block:: yaml - :class: copyable +.. important:: - - name: MINIO_LOG_QUERY_AUTH_TOKEN - valueFrom: - secretKeyRef: - key: MINIO_LOG_QUERY_AUTH_TOKEN - name: -log-secret - - name: MINIO_LOG_QUERY_URL - value: http://-log-search-api:8080 - - name: MINIO_PROMETHEUS_JOB_ID - value: minio-job - - name: MINIO_PROMETHEUS_URL - value: http://-prometheus-hl-svc:9001 + Operator 6.0.0 deprecates the MinIO Operator Console and removes the related resources from the MinIO Operator CRD. - - Replace ```` in the ``name`` or ``value`` lines with the name of your tenant. + MinIO does not remove existing Operator Console services, pods, or deployments as per Kubernetes standard procedure. + You can remove these resources at your own discretion. -Upgrade Operator to |operator-version-stable| -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Use either Kustomization or Helm for managing Tenants moving forward. .. tab-set:: .. tab-item:: Upgrade using Kustomize The following procedure upgrades the MinIO Operator using Kustomize. + For deployments using Operator 5.0.0 through 5.0.14, follow the :ref:`minio-k8s-upgrade-minio-operator-to-5.0.15` procedure before performing this upgrade. - For Operator versions 4.5.8 to 5.0.14 installed with the MinIO Kubernetes Plugin, follow the Kustomize instructions to upgrade to 5.0.15 or later. If you installed the Operator using :ref:`Helm `, use the :guilabel:`Upgrade using Helm` instructions instead. - #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. - - Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. - Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. - See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. - - #. Verify the existing Operator installation. - Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. - - If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. + .. container:: procedure - You can verify the currently installed Operator version by retrieving the object specification for an operator pod in the namespace. - The following example uses the ``jq`` tool to filter the necessary information from ``kubectl``: + #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. - .. code-block:: shell - :class: copyable + Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. + Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. + See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. - kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' + #. Verify the existing Operator installation. + Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. - The output resembles the following: + If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. - .. code-block:: json - :emphasize-lines: 8-10 - :substitutions: + You can verify the currently installed Operator version by retrieving the object specification for an operator pod in the namespace. + The following example uses the ``jq`` tool to filter the necessary information from ``kubectl``: - { - "env": [ - { - "name": "CLUSTER_DOMAIN", - "value": "cluster.local" - } - ], - "image": "minio/operator:v|operator-version-stable|", - "imagePullPolicy": "IfNotPresent", - "name": "minio-operator" - } + .. code-block:: shell + :class: copyable - If your local host does not have the ``jq`` utility installed, you can run the first part of the command and locate the ``spec.containers`` section of the output. + kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' - #. Upgrade Operator with Kustomize + The output resembles the following: - The following command upgrades Operator to version |operator-version-stable|: + .. code-block:: json + :emphasize-lines: 8-10 + :substitutions: - .. code-block:: shell - :class: copyable + { + "env": [ + { + "name": "CLUSTER_DOMAIN", + "value": "cluster.local" + } + ], + "image": "minio/operator:v5.0.15", + "imagePullPolicy": "IfNotPresent", + "name": "minio-operator" + } - kubectl apply -k github.com/minio/operator + If your local host does not have the ``jq`` utility installed, you can run the first part of the command and locate the ``spec.containers`` section of the output. - In the sample output below, ``configured`` indicates where a new change was applied from the updated CRD: + #. Upgrade Operator with Kustomize - .. code-block:: shell + The following command upgrades Operator to version |operator-version-stable|: - namespace/minio-operator configured - customresourcedefinition.apiextensions.k8s.io/miniojobs.job.min.io configured - customresourcedefinition.apiextensions.k8s.io/policybindings.sts.min.io configured - customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io configured - serviceaccount/console-sa unchanged - serviceaccount/minio-operator unchanged - clusterrole.rbac.authorization.k8s.io/console-sa-role unchanged - clusterrole.rbac.authorization.k8s.io/minio-operator-role unchanged - clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding unchanged - clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding unchanged - configmap/console-env unchanged - secret/console-sa-secret configured - service/console unchanged - service/operator unchanged - service/sts unchanged - deployment.apps/console configured - deployment.apps/minio-operator configured + .. code-block:: shell + :class: copyable + kubectl apply -k github.com/minio/operator - #. Validate the Operator upgrade + In the sample output below, ``configured`` indicates where a new change was applied from the updated CRD: - You can check the new Operator version with the same ``kubectl`` command used previously: + .. code-block:: shell - .. code-block:: shell - :class: copyable + namespace/minio-operator unchanged + customresourcedefinition.apiextensions.k8s.io/miniojobs.job.min.io configured + customresourcedefinition.apiextensions.k8s.io/policybindings.sts.min.io configured + customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io configured + serviceaccount/minio-operator unchanged + clusterrole.rbac.authorization.k8s.io/minio-operator-role configured + clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding unchanged + service/operator unchanged + service/sts unchanged + deployment.apps/minio-operator configured - kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' + #. Validate the Operator upgrade - #. *(Optional)* Connect to the Operator Console + You can check the new Operator version with the same ``kubectl`` command used previously: - .. include:: /includes/common/common-k8s-connect-operator-console-no-plugin.rst - - #. Retrieve the Operator Console JWT for login - - .. include:: /includes/common/common-k8s-operator-console-jwt.rst + .. code-block:: shell + :class: copyable + kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers' .. tab-item:: Upgrade using Helm @@ -219,79 +128,82 @@ Upgrade Operator to |operator-version-stable| If you installed the Operator using Kustomize, use the :guilabel:`Upgrade using Kustomize` instructions instead. - #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. + .. container:: procedure + + #. *(Optional)* Update each MinIO Tenant to the latest stable MinIO Version. - Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. - Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. - See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. + Upgrading MinIO regularly ensures your Tenants have the latest features and performance improvements. + Test upgrades in a lower environment such as a Dev or QA Tenant, before applying to your production Tenants. + See :ref:`minio-k8s-upgrade-minio-tenant` for a procedure on upgrading MinIO Tenants. - #. Verify the existing Operator installation. + #. Verify the existing Operator installation. - Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. + Use ``kubectl get all -n minio-operator`` to verify the health and status of all Operator pods and services. - If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. + If you installed the Operator to a custom namespace, specify that namespace as ``-n ``. - Use the ``helm list`` command to view the installed charts in the namespace: + Use the ``helm list`` command to view the installed charts in the namespace: - .. code-block:: shell - :class: copyable + .. code-block:: shell + :class: copyable - helm list -n minio-operator + helm list -n minio-operator - The result should resemble the following: + The result should resemble the following: - .. code-block:: shell - :class: copyable + .. code-block:: shell + :class: copyable - NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION - operator minio-operator 1 2023-11-01 15:49:54.539724775 -0400 EDT deployed operator-5.0.x v5.0.x + NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION + operator minio-operator 1 2023-11-01 15:49:54.539724775 -0400 EDT deployed operator-5.0.x v5.0.x - #. Update the Operator Repository + #. Update the Operator Repository - Use ``helm repo update minio-operator`` to update the MinIO Operator repo. - If you set a different alias for the MinIO Operator repository, specify that in the command instead of ``minio-operator``. - You can use ``helm repo list`` to review your installed repositories. + Use ``helm repo update minio-operator`` to update the MinIO Operator repo. + If you set a different alias for the MinIO Operator repository, specify that in the command instead of ``minio-operator``. + You can use ``helm repo list`` to review your installed repositories. - Use ``helm search`` to check the latest available chart version after updating the Operator Repo: + Use ``helm search`` to check the latest available chart version after updating the Operator Repo: - .. code-block:: shell - :class: copyable + .. code-block:: shell + :class: copyable - helm search repo minio-operator + helm search repo minio-operator - The response should resemble the following: + The response should resemble the following: - .. code-block:: shell - :class: copyable - :substitutions: + .. code-block:: shell + :class: copyable + :substitutions: - NAME CHART VERSION APP VERSION DESCRIPTION - minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator - minio-operator/operator |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator - minio-operator/tenant |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator + NAME CHART VERSION APP VERSION DESCRIPTION + minio-operator/minio-operator 4.3.7 v4.3.7 A Helm chart for MinIO Operator + minio-operator/operator |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator + minio-operator/tenant |operator-version-stable| v|operator-version-stable| A Helm chart for MinIO Operator - The ``minio-operator/minio-operator`` is a legacy chart and should **not** be installed under normal circumstances. + The ``minio-operator/minio-operator`` is a legacy chart and should **not** be installed under normal circumstances. - #. Run ``helm upgrade`` + #. Run ``helm upgrade`` - Helm uses the latest chart to upgrade the MinIO Operator: + Helm uses the latest chart to upgrade the MinIO Operator: - .. code-block:: shell - :class: copyable + .. code-block:: shell + :class: copyable - helm upgrade -n minio-operator \ - operator minio-operator/operator + helm upgrade -n minio-operator \ + operator minio-operator/operator - If you installed the MinIO Operator to a different namespace, specify that in the ``-n`` argument. + If you installed the MinIO Operator to a different namespace, specify that in the ``-n`` argument. - If you used a different installation name from ``operator``, replace the value above with the installation name. + If you used a different installation name from ``operator``, replace the value above with the installation name. - The command results should return success with a bump in the ``REVISION`` value. + The command results should return success with a bump in the ``REVISION`` value. - #. Validate the Operator upgrade + #. Validate the Operator upgrade - .. include:: /includes/common/common-k8s-connect-operator-console-no-plugin.rst + You can check the new Operator version with the same ``kubectl`` command used previously: - #. Retrieve the Operator Console JWT for login + .. code-block:: shell + :class: copyable - .. include:: /includes/common/common-k8s-operator-console-jwt.rst + kubectl get pod -l 'name=minio-operator' -n minio-operator -o json | jq '.items[0].spec.containers'