From e529618d6335dcf5cd3bf703c640a4c192377adf Mon Sep 17 00:00:00 2001 From: lemonprogis Date: Fri, 2 Aug 2024 15:55:48 -0500 Subject: [PATCH 1/8] Adding a new prereq file around deployments --- docs/deployment/prerequisites.md | 53 ++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 docs/deployment/prerequisites.md diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md new file mode 100644 index 000000000..2662816c2 --- /dev/null +++ b/docs/deployment/prerequisites.md @@ -0,0 +1,53 @@ +--- +title: UDS Prerequisites +type: docs +weight: 4 +--- + +## UDS Prerequisites (by components) + +These components are what comprise `UDS Core`. Below each, you will find any prerequisites that have been discovered through different deployment scenarios across different environments. + +#### Istio + +Solution is to create file `/etc/modules-load.d/istio-iptables.conf` with this content: + +```bash +# These modules need to be loaded on boot so that Istio (as required by +# UDS Core) runs properly. +# +# See also: https://github.com/istio/istio/issues/23009 + +br_netfilter +nf_nat +xt_REDIRECT +xt_owner +iptable_nat +iptable_mangle +iptable_filter +``` + +```bash +sudo systemctl stop firewalld +sudo systemctl disable firewalld +``` + +#### Pepr + +#### Metrics Server + +#### Keycloak + +#### Neuvector + +#### Loki + +#### Prometheus + +#### Promtail + +#### Grafana + +#### Authservice + +#### Velero From 5c6b9a2ccbab191c611deb75d342f8607756c6f4 Mon Sep 17 00:00:00 2001 From: lemonprogis Date: Wed, 21 Aug 2024 19:47:26 -0500 Subject: [PATCH 2/8] swapped ideas --- docs/deployment/prerequisites.md | 38 +++++++++++++++++++------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index 2662816c2..26d7bc03f 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -4,12 +4,13 @@ type: docs weight: 4 --- -## UDS Prerequisites (by components) +## UDS installation prerequisites -These components are what comprise `UDS Core`. Below each, you will find any prerequisites that have been discovered through different deployment scenarios across different environments. +`UDS Core` could run in any [Kuberentes](https://kubernetes.io/) setup, but sometimes customizations are needed based on environemnts. This is an attempt to document and link to relevant information to aid in setting up your [Kuberentes](https://kubernetes.io/) environment to ensure a successful `UDS Core` installation. -#### Istio +### RHEL +#### *ISTIO related changes* Solution is to create file `/etc/modules-load.d/istio-iptables.conf` with this content: ```bash @@ -32,22 +33,29 @@ sudo systemctl stop firewalld sudo systemctl disable firewalld ``` -#### Pepr -#### Metrics Server +Other Istio documentation: +* https://istio.io/latest/docs/ops/deployment/platform-requirements/ -#### Keycloak +### RKE2 +* https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking +* https://docs.rke2.io/install/requirements +* https://github.com/defenseunicorns/uds-rke2-image-builder/blob/main/packer/scripts/os-prep.sh -#### Neuvector -#### Loki +### K3S +* https://docs.k3s.io/installation/requirements#operating-systems -#### Prometheus -#### Promtail -#### Grafana - -#### Authservice - -#### Velero +#### UDS Core components +* UDS Operator +* [Istio](https://istio.io/latest/docs/ops/deployment/platform-requirements) +* [Keycloak](https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/) +* Neuvector +* Loki +* Prometheus +* Promtail +* Grafana +* Authservice +* Velero From ed11fb631060929b0cd2c038478d3db6cbc443d1 Mon Sep 17 00:00:00 2001 From: lemonprogis Date: Thu, 22 Aug 2024 11:54:10 -0500 Subject: [PATCH 3/8] updates around metrics server --- docs/deployment/prerequisites.md | 47 ++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index 26d7bc03f..3c92da868 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -33,29 +33,40 @@ sudo systemctl stop firewalld sudo systemctl disable firewalld ``` - -Other Istio documentation: -* https://istio.io/latest/docs/ops/deployment/platform-requirements/ - ### RKE2 -* https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking -* https://docs.rke2.io/install/requirements -* https://github.com/defenseunicorns/uds-rke2-image-builder/blob/main/packer/scripts/os-prep.sh +* [Installation requirements](https://docs.rke2.io/install/requirements) +* [Firewalld network conflicts](https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking) +* [Disabling components, such as Ingress which clashes with istio](https://docs.rke2.io/advanced#disabling-server-charts) +* [Defense Unicorns os prep script for rke2](https://github.com/defenseunicorns/uds-rke2-image-builder/blob/main/packer/scripts/os-prep.sh) ### K3S -* https://docs.k3s.io/installation/requirements#operating-systems +* [OS requirements](https://docs.k3s.io/installation/requirements#operating-systems) #### UDS Core components -* UDS Operator -* [Istio](https://istio.io/latest/docs/ops/deployment/platform-requirements) -* [Keycloak](https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/) -* Neuvector -* Loki -* Prometheus -* Promtail -* Grafana -* Authservice -* Velero +#### UDS Operator +#### Istio +* [Platform requirements](https://istio.io/latest/docs/ops/deployment/platform-requirements/) +#### Keycloak +* [Configuration guide](https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/) +#### Neuvector +#### Loki +#### Prometheus +#### Promtail +#### Grafana +#### Authservice +#### Velero +#### Metrics Server +* Optional component and can be added if needed. Most of the provided managed clsuters will provide you a metric server. +```yaml +... +- name: uds-core + repository: ghcr.io/defenseunicorns/packages/private/uds/core + ref: 0.25.2-unicorn + optionalComponents: + - metrics-server +... +``` + From 012b3800e3c09ca73f12dc26291b356b3effca8f Mon Sep 17 00:00:00 2001 From: lemonprogis Date: Thu, 22 Aug 2024 12:00:22 -0500 Subject: [PATCH 4/8] some clean up --- docs/deployment/prerequisites.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index 3c92da868..bb7700ecd 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -9,7 +9,7 @@ weight: 4 `UDS Core` could run in any [Kuberentes](https://kubernetes.io/) setup, but sometimes customizations are needed based on environemnts. This is an attempt to document and link to relevant information to aid in setting up your [Kuberentes](https://kubernetes.io/) environment to ensure a successful `UDS Core` installation. ### RHEL - +--- #### *ISTIO related changes* Solution is to create file `/etc/modules-load.d/istio-iptables.conf` with this content: @@ -33,7 +33,8 @@ sudo systemctl stop firewalld sudo systemctl disable firewalld ``` -### RKE2 +### RKE2 +--- * [Installation requirements](https://docs.rke2.io/install/requirements) * [Firewalld network conflicts](https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking) * [Disabling components, such as Ingress which clashes with istio](https://docs.rke2.io/advanced#disabling-server-charts) @@ -41,11 +42,13 @@ sudo systemctl disable firewalld ### K3S +--- * [OS requirements](https://docs.k3s.io/installation/requirements#operating-systems) -#### UDS Core components +### UDS Core components +--- #### UDS Operator #### Istio * [Platform requirements](https://istio.io/latest/docs/ops/deployment/platform-requirements/) From 5f19a3be1a9f9ace1817159b032193a8c65e583d Mon Sep 17 00:00:00 2001 From: Ed <1834551+lemonprogis@users.noreply.github.com> Date: Mon, 26 Aug 2024 16:14:19 -0500 Subject: [PATCH 5/8] Update docs/deployment/prerequisites.md Co-authored-by: Micah Nagel --- docs/deployment/prerequisites.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index bb7700ecd..f53dd69ed 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -6,7 +6,7 @@ weight: 4 ## UDS installation prerequisites -`UDS Core` could run in any [Kuberentes](https://kubernetes.io/) setup, but sometimes customizations are needed based on environemnts. This is an attempt to document and link to relevant information to aid in setting up your [Kuberentes](https://kubernetes.io/) environment to ensure a successful `UDS Core` installation. +`UDS Core` can run in any CNCF conformant [Kubernetes](https://www.cncf.io/training/certification/software-conformance/) setup, but sometimes customizations are needed based on environments. This is an attempt to document and link to relevant information to aid in setting up your Kubernetes environment and hosts to ensure a successful `UDS Core` installation. ### RHEL --- From 9ebd782448c532b704a9a01ec3376dc2f46c15c8 Mon Sep 17 00:00:00 2001 From: Ed <1834551+lemonprogis@users.noreply.github.com> Date: Mon, 26 Aug 2024 16:15:20 -0500 Subject: [PATCH 6/8] Update docs/deployment/prerequisites.md Co-authored-by: Micah Nagel --- docs/deployment/prerequisites.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index f53dd69ed..c080dd07c 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -62,7 +62,7 @@ sudo systemctl disable firewalld #### Authservice #### Velero #### Metrics Server -* Optional component and can be added if needed. Most of the provided managed clsuters will provide you a metric server. +* Optional component and can be added if needed. Most of the provided managed clusters will provide you a metric server. ```yaml ... - name: uds-core From ebca139655ab14ad7c0a3791a658601864ac7808 Mon Sep 17 00:00:00 2001 From: Micah Nagel Date: Tue, 3 Sep 2024 08:47:43 -0600 Subject: [PATCH 7/8] docs: updates based on PR feedback --- docs/deployment/distribution-support.md | 10 +- docs/deployment/prerequisites.md | 116 ++++++++++++++---------- 2 files changed, 74 insertions(+), 52 deletions(-) diff --git a/docs/deployment/distribution-support.md b/docs/deployment/distribution-support.md index 5910d2cd1..268366559 100644 --- a/docs/deployment/distribution-support.md +++ b/docs/deployment/distribution-support.md @@ -12,8 +12,8 @@ UDS Core is a versatile software baseline designed to operate effectively across - **Compatible:** Kubernetes distributions listed under this category may not have undergone extensive testing in UDS Core's CI environments. While UDS Core may be compatible on these distributions, users should exercise caution and be prepared for potential compatibility issues or limitations. -| Distribution | Category | Support Level | -| --------------- | ---------------------- | --------------------------------------------------------------------------------------------------------- | -| K3d, Amazon EKS | Tested | Supported Kubernetes distributions undergoing testing in CI environments. | -| RKE2 | Tested | Supported Kubernetes distribution tested in production environments other than CI. | -| Other | Untested/Unknown state | Compatible Kubernetes distributions that are not explicitly tested, documented, or supported by UDS Core. | +| Distribution | Category | Support Level | +| ------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------- | +| K3d/K3s, Amazon EKS | Tested | Supported Kubernetes distributions undergoing testing in CI environments. | +| RKE2 | Tested | Supported Kubernetes distribution tested in production environments other than CI. | +| Other | Untested/Unknown state | Compatible Kubernetes distributions that are not explicitly tested, documented, or supported by UDS Core. | diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index c080dd07c..c5645470d 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -4,65 +4,88 @@ type: docs weight: 4 --- -## UDS installation prerequisites +## UDS Installation Prerequisites `UDS Core` can run in any CNCF conformant [Kubernetes](https://www.cncf.io/training/certification/software-conformance/) setup, but sometimes customizations are needed based on environments. This is an attempt to document and link to relevant information to aid in setting up your Kubernetes environment and hosts to ensure a successful `UDS Core` installation. -### RHEL ---- -#### *ISTIO related changes* -Solution is to create file `/etc/modules-load.d/istio-iptables.conf` with this content: - -```bash -# These modules need to be loaded on boot so that Istio (as required by -# UDS Core) runs properly. -# -# See also: https://github.com/istio/istio/issues/23009 - -br_netfilter -nf_nat -xt_REDIRECT -xt_owner -iptable_nat -iptable_mangle -iptable_filter -``` +When running Kubernetes on any type of host it is important to ensure you are following the upstream documentation from the Kubernetes distribution regarding prerequisites. A few links to upstream documentation are provided below for convenience. -```bash -sudo systemctl stop firewalld -sudo systemctl disable firewalld -``` +### Cluster Requirements -### RKE2 ---- -* [Installation requirements](https://docs.rke2.io/install/requirements) -* [Firewalld network conflicts](https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking) -* [Disabling components, such as Ingress which clashes with istio](https://docs.rke2.io/advanced#disabling-server-charts) -* [Defense Unicorns os prep script for rke2](https://github.com/defenseunicorns/uds-rke2-image-builder/blob/main/packer/scripts/os-prep.sh) +#### RKE2 +- [General installation requirements](https://docs.rke2.io/install/requirements) +- [Disabling Firewalld to prevent networking conflicts](https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking) +- [Modifying NetworkManager to prevent CNI conflicts](https://docs.rke2.io/known_issues#networkmanager) +- [Additional Known Issues](https://docs.rke2.io/known_issues) ### K3S ---- -* [OS requirements](https://docs.k3s.io/installation/requirements#operating-systems) +- [General installation requirements](https://docs.k3s.io/installation/requirements) +- [Known Issues](https://docs.k3s.io/known-issues) +### EKS + +- [General installation requirements](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) +- [Troubleshooting Guide](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html) + +### UDS Core Requirements + +#### Default Storage Class + +Several UDS Core components require persistent volumes that will be provisioned using the default storage class via dynamic volume provisioning. Ensure that your cluster includes a default storage class prior to deploying. You can validate by running the below command (see example output which includes `(default)` next to the `local-path` storage class): + +```console +❯ kubectl get storageclass +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 55s +``` + +#### Network Policy Support + +The UDS Operator will dynamically provision network policies to secure traffic between components in UDS Core. To ensure these are effective, validate that your CNI supports enforcing network policies. In addition, UDS Core makes use of some CIDR based policies for communication with the KubeAPI server. If you are using Cilium, support for node addressability with CIDR based policies must be enabled with a [feature flag](https://docs.cilium.io/en/stable/security/policy/language/#selecting-nodes-with-cidr-ipblock). + +#### Istio + +Istio requires a number of kernel modules to be loaded for full functionality. The below is a script that will ensure these modules are loaded and persisted across reboots (see also Istio's [upstream requirements list](https://istio.io/latest/docs/ops/deployment/platform-requirements/)): + +```console +modules=("br_netfilter" "xt_REDIRECT" "xt_owner" "xt_statistic" "iptable_mangle" "iptable_nat" "xt_conntrack" "xt_tcpudp") +for module in "${modules[@]}"; do + modprobe "$module" + echo "$module" >> "/etc/modules-load.d/istio-modules.conf" +done +``` + +In addition, to run Istio ingress gateways (part of Core) you will need to ensure your cluster supports dynamic load balancer provisioning when services of type LoadBalancer are created. Typically in cloud environments this is handled using a cloud provider's controller (example: [AWS LB Controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller)). When deploying on-prem, this is commonly done by using a "bare metal" load balancer provisioner like [MetalLB](https://metallb.universe.tf/) or [kube-vip](https://kube-vip.io/). Certain distributions may also include ingress controllers that you will want to disable as they may conflict with Istio (example: RKE2 includes ingress-nginx). + +#### NeuVector + +NeuVector historically has functioned best when the host is using cgroup v2. Cgroup v2 is enabled by default on many modern Linux distributions, but you may need to enable it depending on your operating system. Enabling this tends to be OS specific, so you will need to evaluate this for your specific hosts. -### UDS Core components ---- -#### UDS Operator -#### Istio -* [Platform requirements](https://istio.io/latest/docs/ops/deployment/platform-requirements/) -#### Keycloak -* [Configuration guide](https://www.keycloak.org/keycloak-benchmark/kubernetes-guide/latest/) -#### Neuvector -#### Loki -#### Prometheus #### Promtail -#### Grafana -#### Authservice -#### Velero + +In order to ensure that Promtail is able to scrape the necessary logs concurrently you may need to adjust some kernel parameters for your hosts. The below is a script to adjust these parameters to suitable values and ensure they are persisted across reboots: + +```console +declare -A sysctl_settings +sysctl_settings["fs.nr_open"]=13181250 +sysctl_settings["fs.inotify.max_user_instances"]=1024 +sysctl_settings["fs.inotify.max_user_watches"]=1048576 +sysctl_settings["fs.file-max"]=13181250 + +for key in "${!sysctl_settings[@]}"; do + value="${sysctl_settings[$key]}" + sysctl -w "$key=$value" + echo "$key=$value" > "/etc/sysctl.d/$key.conf" +done +sysctl -p +``` + #### Metrics Server -* Optional component and can be added if needed. Most of the provided managed clusters will provide you a metric server. + +Metrics server is provided as an optional component in UDS Core and can be enabled if needed. For distros where metrics-server is already provided, ensure that you do NOT enable metrics-server. See the below as an example for enabling metrics-server if your cluster does not include it. + ```yaml ... - name: uds-core @@ -72,4 +95,3 @@ sudo systemctl disable firewalld - metrics-server ... ``` - From 0c2bfa466987283434df3ed61043407f1b610bdd Mon Sep 17 00:00:00 2001 From: Micah Nagel Date: Tue, 3 Sep 2024 14:20:16 -0600 Subject: [PATCH 8/8] docs: cleanup/feedback from @UnicornChance --- docs/deployment/prerequisites.md | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/deployment/prerequisites.md b/docs/deployment/prerequisites.md index c5645470d..fdbc2fb3e 100644 --- a/docs/deployment/prerequisites.md +++ b/docs/deployment/prerequisites.md @@ -1,36 +1,43 @@ --- -title: UDS Prerequisites +title: UDS Core Prerequisites type: docs weight: 4 --- -## UDS Installation Prerequisites +## UDS Core Prerequisites -`UDS Core` can run in any CNCF conformant [Kubernetes](https://www.cncf.io/training/certification/software-conformance/) setup, but sometimes customizations are needed based on environments. This is an attempt to document and link to relevant information to aid in setting up your Kubernetes environment and hosts to ensure a successful `UDS Core` installation. - -When running Kubernetes on any type of host it is important to ensure you are following the upstream documentation from the Kubernetes distribution regarding prerequisites. A few links to upstream documentation are provided below for convenience. +`UDS Core` can run in any [CNCF conformant Kubernetes distribution](https://www.cncf.io/training/certification/software-conformance/), but sometimes customizations are needed based on environments. This is an attempt to document and link to relevant information to aid in setting up your Kubernetes environment and hosts to ensure a successful `UDS Core` installation. ### Cluster Requirements +When running Kubernetes on any type of host it is important to ensure you are following the upstream documentation from the Kubernetes distribution regarding prerequisites. A few links to upstream documentation are provided below for convenience. + #### RKE2 - [General installation requirements](https://docs.rke2.io/install/requirements) - [Disabling Firewalld to prevent networking conflicts](https://docs.rke2.io/known_issues#firewalld-conflicts-with-default-networking) - [Modifying NetworkManager to prevent CNI conflicts](https://docs.rke2.io/known_issues#networkmanager) -- [Additional Known Issues](https://docs.rke2.io/known_issues) +- [Known Issues](https://docs.rke2.io/known_issues) -### K3S +#### K3S - [General installation requirements](https://docs.k3s.io/installation/requirements) - [Known Issues](https://docs.k3s.io/known-issues) -### EKS +#### EKS - [General installation requirements](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) - [Troubleshooting Guide](https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html) +#### AKS + +- [General installation requirements](https://learn.microsoft.com/en-us/azure/well-architected/service-guides/azure-kubernetes-service) +- [Troubleshooting Guide](https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/welcome-azure-kubernetes) + ### UDS Core Requirements +The below are specific requirements for running UDS Core. Some of them are tied to the entire stack of UDS Core and some are more specific to certain components. If you encounter issues with a particular component of core, this can be a good list to check to validate you met all the prerequisite requirements for that specific application. + #### Default Storage Class Several UDS Core components require persistent volumes that will be provisioned using the default storage class via dynamic volume provisioning. Ensure that your cluster includes a default storage class prior to deploying. You can validate by running the below command (see example output which includes `(default)` next to the `local-path` storage class): @@ -47,7 +54,7 @@ The UDS Operator will dynamically provision network policies to secure traffic b #### Istio -Istio requires a number of kernel modules to be loaded for full functionality. The below is a script that will ensure these modules are loaded and persisted across reboots (see also Istio's [upstream requirements list](https://istio.io/latest/docs/ops/deployment/platform-requirements/)): +Istio requires a number of kernel modules to be loaded for full functionality. The below is a script that will ensure these modules are loaded and persisted across reboots (see also Istio's [upstream requirements list](https://istio.io/latest/docs/ops/deployment/platform-requirements/)). Ideally this script is used as part of an image build or cloud-init process on each node. ```console modules=("br_netfilter" "xt_REDIRECT" "xt_owner" "xt_statistic" "iptable_mangle" "iptable_nat" "xt_conntrack" "xt_tcpudp") @@ -65,7 +72,7 @@ NeuVector historically has functioned best when the host is using cgroup v2. Cgr #### Promtail -In order to ensure that Promtail is able to scrape the necessary logs concurrently you may need to adjust some kernel parameters for your hosts. The below is a script to adjust these parameters to suitable values and ensure they are persisted across reboots: +In order to ensure that Promtail is able to scrape the necessary logs concurrently you may need to adjust some kernel parameters for your hosts. The below is a script that can be used to adjust these parameters to suitable values and ensure they are persisted across reboots. Ideally this script is used as part of an image build or cloud-init process on each node. ```console declare -A sysctl_settings