You are being redirected to the OpenNESS Docs.
diff --git a/doc/cloud-adapters/openness_baiducloud.md b/doc/cloud-adapters/openness_baiducloud.md index 57619722..5083d4ab 100644 --- a/doc/cloud-adapters/openness_baiducloud.md +++ b/doc/cloud-adapters/openness_baiducloud.md @@ -1,6 +1,6 @@ ```text SPDX-License-Identifier: Apache-2.0 -Copyright (c) 2019 Intel Corporation +Copyright (c) 2019-2021 Intel Corporation ``` # OpenNESS Integration with Baidu OpenEdge @@ -322,7 +322,7 @@ The scripts can be found in the release package with the subfolder name `setup_b └── measure_rtt_openedge.py ``` -Before running the scripts, install python3.6 and paho mqtt on a CentOS\* Linux\* machine, where the recommended version is CentOS Linux release 7.8.2003 (Core). +Before running the scripts, install python3.6 and paho mqtt on a CentOS\* Linux\* machine, where the recommended version is CentOS Linux release 7.9.2009 (Core). The following are recommended install commands: ```docker diff --git a/doc/devkits/openness-azure-devkit.md b/doc/devkits/openness-azure-devkit.md index 889f9af5..0850bc74 100644 --- a/doc/devkits/openness-azure-devkit.md +++ b/doc/devkits/openness-azure-devkit.md @@ -14,4 +14,4 @@ for automated depoyment, and supports deployment using Porter. It enables cloud ## Getting Started Following document contains steps for quick deployment on Azure: -* [openness-experience-kits/cloud/README.md: Deployment and setup guide](https://github.com/open-ness/openness-experience-kits/blob/master/cloud/README.md) +* [converged-edge-experience-kits/cloud/README.md: Deployment and setup guide](https://github.com/open-ness/converged-edge-experience-kits/blob/master/cloud/README.md) diff --git a/doc/flavors.md b/doc/flavors.md index cbb84c1b..160c8c4d 100644 --- a/doc/flavors.md +++ b/doc/flavors.md @@ -3,56 +3,82 @@ SPDX-License-Identifier: Apache-2.0 Copyright (c) 2020 Intel Corporation ``` -- [OpenNESS Deployment Flavors](#openness-deployment-flavors) - - [CERA Minimal Flavor](#cera-minimal-flavor) - - [CERA Access Edge Flavor](#cera-access-edge-flavor) - - [CERA Media Analytics Flavor](#cera-media-analytics-flavor) - - [CERA Media Analytics Flavor with VCAC-A](#cera-media-analytics-flavor-with-vcac-a) - - [CERA CDN Transcode Flavor](#cera-cdn-transcode-flavor) - - [CERA CDN Caching Flavor](#cera-cdn-caching-flavor) - - [CERA Core Control Plane Flavor](#cera-core-control-plane-flavor) - - [CERA Core User Plane Flavor](#cera-core-user-plane-flavor) - - [CERA Untrusted Non3gpp Access Flavor](#cera-untrusted-non3gpp-access-flavor) - - [CERA Near Edge Flavor](#cera-near-edge-flavor) - - [CERA 5G On-Prem Flavor](#cera-5g-on-prem-flavor) - - [Reference Service Mesh](#reference-service-mesh) - - [Central Orchestrator Flavor](#central-orchestrator-flavor) - # OpenNESS Deployment Flavors - -This document introduces the supported deployment flavors that are deployable through OpenNESS Experience Kits (OEKs. +This document introduces the supported deployment flavors that are deployable through the Converged Edge Experience Kits (CEEK). + +- [CERA Minimal Flavor](#cera-minimal-flavor) +- [CERA Access Edge Flavor](#cera-access-edge-flavor) +- [CERA Media Analytics Flavor](#cera-media-analytics-flavor) +- [CERA Media Analytics Flavor with VCAC-A](#cera-media-analytics-flavor-with-vcac-a) +- [CERA CDN Transcode Flavor](#cera-cdn-transcode-flavor) +- [CERA CDN Caching Flavor](#cera-cdn-caching-flavor) +- [CERA Core Control Plane Flavor](#cera-core-control-plane-flavor) +- [CERA Core User Plane Flavor](#cera-core-user-plane-flavor) +- [CERA Untrusted Non3gpp Access Flavor](#cera-untrusted-non3gpp-access-flavor) +- [CERA Near Edge Flavor](#cera-near-edge-flavor) +- [CERA 5G On-Prem Flavor](#cera-5g-on-prem-flavor) +- [CERA 5G Central Office Flavor](#cera-5g-central-office-flavor) +- [Central Orchestrator Flavor](#central-orchestrator-flavor) +- [CERA SD-WAN Edge Flavor](#cera-sd-wan-edge-flavor) +- [CERA SD-WAN Hub Flavor](#cera-sd-wan-hub-flavor) ## CERA Minimal Flavor The pre-defined *minimal* deployment flavor provisions the minimal set of configurations for bringing up the OpenNESS network edge deployment. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run the OEK deployment script: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `minimal` + ```yaml + --- + all: + vars: + cluster_name: minimal_cluster + flavor: minimal + ... + ``` +3. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f minimal + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: * Node feature discovery -* The default Kubernetes CNI: `kube-ovn` +* The default Kubernetes CNI: `calico` * Telemetry +To customize this flavor we recommend creating additional file in converged-edge-experience-kits that will override any variables used in previous configuration. This file should be placed in location: `converged-edge-experience-kits/inventory/default/group_vars/all` and filenames should start with number greater than highest value currently present (e.g. `40-overrides.yml`). + ## CERA Access Edge Flavor The pre-defined *flexran* deployment flavor provisions an optimized system configuration for vRAN workloads on Intel® Xeon® platforms. It also provisions for deployment of Intel® FPGA Programmable Acceleration Card (Intel® FPGA PAC) N3000 tools and components to enable offloading for the acceleration of FEC (Forward Error Correction) to the FPGA. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). 2. Configure the flavor file to reflect desired deployment. - - Configure the CPUs selected for isolation and OS/K8s processes from command line in files [controller_group.yml](https://github.com/open-ness/openness-experience-kits/blob/master/flavors/flexran/controller_group.yml) and [edgenode_group.yml](https://github.com/open-ness/openness-experience-kits/blob/master/flavors/flexran/edgenode_group.yml) - please note that in single node mode the edgenode_group.yml is used to configure the CPU isolation. - - Configure the amount of CPUs reserved for K8s and OS from K8s level with `reserved_cpu` flag in [all.yml](https://github.com/open-ness/openness-experience-kits/blob/master/flavors/flexran/all.yml) file. - - Configure whether the FPGA or eASIC support for FEC is desired or both in [all.yml](https://github.com/open-ness/openness-experience-kits/blob/master/flavors/flexran/all.yml) file. - -3. Run OEK deployment script: + - Configure the CPUs selected for isolation and OS/K8s processes from command line in files [controller_group.yml](https://github.com/open-ness/ido-converged-edge-experience-kits/blob/master/flavors/flexran/controller_group.yml) and [edgenode_group.yml](https://github.com/open-ness/ido-converged-edge-experience-kits/blob/master/flavors/flexran/edgenode_group.yml) - please note that in single node mode the edgenode_group.yml is used to configure the CPU isolation. + - Configure which CPUs are to be reserved for K8s and OS from K8s level with `reserved_cpu` flag in [all.yml](https://github.com/open-ness/ido-converged-edge-experience-kits/blob/master/flavors/flexran/all.yml) file. + - Configure whether the FPGA or eASIC support for FEC is desired or both in [all.yml](https://github.com/open-ness/ido-converged-edge-experience-kits/blob/master/flavors/flexran/all.yml) file. + +3. Provide necessary files: + - Create the `ido-converged-edge-experience-kits/ceek/biosfw` directory and copy the `syscfg_package.zip` file to the directory (can be disabled with `ne_biosfw_enable` flag). + - Create the `ido-converged-edge-experience-kits/ceek/opae_fpga` directory and copy the OPAE_SDK_1.3.7-5_el7.zip to the directory (can be disabled with `ne_opae_fpga_enable` flag) + - Create the `ido-converged-edge-experience-kits/ceek/nic_drivers` directory and copy the `ice-1.3.2.tar.gz` and `iavf-4.0.2.tar.gz` files to the directory (can be disabled with `e810_driver_enable` flag). + +4. Update the `inventory.yaml` file by setting the deployment flavor as `flexran` + ```yaml + --- + all: + vars: + cluster_name: flexran_cluster + flavor: flexran + ... + ``` + +5. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f flexran + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: * Node Feature Discovery @@ -62,6 +88,7 @@ This deployment flavor enables the following ingredients: * FPGA remote system update through OPAE * FPGA configuration * eASIC ACC100 configuration +* E810 and IAVF kernel driver update * RT Kernel * Topology Manager * RMD operator @@ -71,18 +98,28 @@ This deployment flavor enables the following ingredients: The pre-defined *media-analytics* deployment flavor provisions an optimized system configuration for media analytics workloads on Intel® Xeon® platforms. It also provisions a set of video analytics services based on the [Video Analytics Serving](https://github.com/intel/video-analytics-serving) for analytics pipeline management and execution. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run the OEK deployment script: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `media-analytics` + ```yaml + --- + all: + vars: + cluster_name: media_analytics_cluster + flavor: media-analytics + ... + ``` +3. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f media-analytics + $ python3 deploy.py ``` > **NOTE:** The video analytics services integrates with the OpenNESS service mesh when the flag `ne_istio_enable: true` is set. > **NOTE:** Kiali management console username can be changed by editing the variable `istio_kiali_username`. By default `istio_kiali_password` is randomly generated and can be retirieved by running `kubectl get secrets/kiali -n istio-system -o json | jq -r '.data.passphrase' | base64 -d` on the Kubernetes controller. +> **NOTE:** Istio deployment can be customized using parameters in the `flavor/media-analytics/all.yaml` (parameters set in the flavor file override default parameters set in `inventory/default/group_vars/all/10-default.yml`). This deployment flavor enables the following ingredients: * Node feature discovery -* The default Kubernetes CNI: `kube-ovn` +* The default Kubernetes CNI: `calico` * Video analytics services * Telemetry * Istio service mesh - conditional @@ -90,32 +127,43 @@ This deployment flavor enables the following ingredients: ## CERA Media Analytics Flavor with VCAC-A -The pre-defined *media-analytics-vca* deployment flavor provisions an optimized system configuration for media analytics workloads leveraging Visual Cloud Accelerator Card – Analytics (VCAC-A) acceleration. It also provisions a set of video analytics services based on the [Video Analytics Serving](https://github.com/intel/video-analytics-serving) for analytics pipeline management and execution. +The pre-defined *media-analytics-vca* deployment flavor provisions an optimized system configuration for media analytics workloads leveraging Visual Cloud Accelerator Card for Analytics (VCAC-A) acceleration. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Add the VCA hostname in the `[edgenode_vca_group]` group in `inventory.ini` file of the OEK, for example: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Add the VCA host name in the `edgenode_vca_group:` group in `inventory.yml` file of the CEEK, e.g: + ```yaml + edgenode_vca_group: + hosts: + vca-node01.openness.org: + ansible_host: 172.16.0.1 + ansible_user: openness ``` - [edgenode_vca_group] - silpixa00400194 + > **NOTE:** The VCA host name should *only* be placed once in the `inventory.yml` file and under the `edgenode_vca_group:` group. + +3. Update the `inventory.yaml` file by setting the deployment flavor as `media-analytics-vca` + ```yaml + --- + all: + vars: + cluster_name: media_analytics_vca_cluster + flavor: media-analytics-vca + ... ``` - > **NOTE:** The VCA host name should *only* be placed once in the `inventory.ini` file and under the `[edgenode_vca_group]` group. - -3. Run the OEK deployment script: +4. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f media-analytics-vca + $ python3 deploy.py ``` -> **NOTE:** At the time of writing this document, *Weave Net*\* is the only supported CNI for network edge deployments involving VCAC-A acceleration. The `weavenet` CNI is automatically selected by the *media-analytics-vca*. -> **NOTE:** The flag `force_build_enable` (default true) supports force build VCAC-A system image (VCAD) by default, it is defined in flavors/media-analytics-vca/all.yml. By setting the flag as false, OEK will not rebuild the image and re-use the last system image built during deployment. If the flag is true, OEK will force build VCA host kernel and node system image which will take several hours. +> **NOTE:** At the time of writing this document, *Weave Net*\* is the only supported CNI for network edge deployments involving VCAC-A acceleration. The `weavenet` CNI is automatically selected by the *media-analytics-vca*. +> **NOTE:** The flag `force_build_enable` (default true) supports force build VCAC-A system image (VCAD) by default, it is defined in flavors/media-analytics-vca/all.yml. By setting the flag as false, CEEK will not rebuild the image and re-use the last system image built during deployment. If the flag is true, CEEK will force build VCA host kernel and node system image which will take several hours. This deployment flavor enables the following ingredients: * Node feature discovery * VPU and GPU device plugins * HDDL daemonset * The `weavenet` Kubernetes CNI -* Video analytics services * Telemetry ## CERA CDN Transcode Flavor @@ -123,15 +171,24 @@ This deployment flavor enables the following ingredients: The pre-defined *cdn-transcode* deployment flavor provisions an optimized system configuration for Content Delivery Network (CDN) transcode sample workloads on Intel® Xeon® platforms. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run the OEK deployment script: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `cdn-transcode` + ```yaml + --- + all: + vars: + cluster_name: cdn_transcode_cluster + flavor: cdn-transcode + ... + ``` +3. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f cdn-transcode + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: * Node feature discovery -* The default Kubernetes CNI: `kube-ovn` +* The default Kubernetes CNI: `calico` * Telemetry ## CERA CDN Caching Flavor @@ -139,10 +196,19 @@ This deployment flavor enables the following ingredients: The pre-defined *cdn-caching* deployment flavor provisions an optimized system configuration for CDN content delivery workloads on Intel® Xeon® platforms. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run the OEK deployment script: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `cdn-caching` + ```yaml + --- + all: + vars: + cluster_name: cdn_caching_cluster + flavor: cdn-caching + ... + ``` +3. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f cdn-caching + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: @@ -157,17 +223,25 @@ The pre-defined Core Control Plane flavor provisions the minimal set of configur The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). - -2. Run the x-OEK deployment script: - ``` - $ ido-openness-experience-kits# deploy_ne.sh -f core-cplane - ``` +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `core-cplane` + ```yaml + --- + all: + vars: + cluster_name: core_cplane_cluster + flavor: core-cplane + ... + ``` +3. Run ido-CEEK deployment script: + ```shell + $ python3 deploy.py + ``` This deployment flavor enables the following ingredients: - Node feature discovery -- The default Kubernetes CNI: kube-ovn +- The default Kubernetes CNIs: calico, sriov - Telemetry - OpenNESS 5G Microservices - OAM(Operation, Administration, Maintenance) and AF(Application Function) on the OpenNESS Controller/K8S Master. @@ -175,7 +249,7 @@ This deployment flavor enables the following ingredients: - Istio service mesh - Kiali management console -> **NOTE:** It is an expectation that the `core-cplane` deployment flavor is done for a setup consisting of *at least one* OpenNESS edge node, i.e: the `inventory.ini` must contain at least one host name under the `edgenode_group` section. +> **NOTE:** It is an expectation that the `core-cplane` deployment flavor is done for a setup consisting of *at least one* OpenNESS edge node, i.e: the `inventory/default/inventory.ini` must contain at least one host name under the `edgenode_group` section. > **NOTE:** For a real deployment with the 5G Core Network Functions the NEF and CNTF can be uninstalled using helm charts. Refer to [OpenNESS using CNCA](applications-onboard/using-openness-cnca.md) @@ -186,17 +260,25 @@ This deployment flavor enables the following ingredients: The pre-defined Core Control Plane flavor provisions the minimal set of configurations for a 5G User Plane Function on Intel® Xeon® platforms. The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). - -2. Run the x-OEK deployment script: - ``` - $ ido-openness-experience-kits# deploy_ne.sh -f core-uplane - ``` +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `core-uplane` + ```yaml + --- + all: + vars: + cluster_name: core_uplane_cluster + flavor: core-uplane + ... + ``` +3. Run ido-CEEK deployment script: + ```shell + $ python3 deploy.py + ``` This deployment flavor enables the following ingredients: - Node feature discovery -- Kubernetes CNI: kube-ovn and SRIOV. +- Kubernetes CNI: calico and SRIOV. - CPU Manager for Kubernetes (CMK) with 4 exclusive cores (1 to 4) and 1 core in shared pool. - Kubernetes Device Plugin - Telemetry @@ -210,13 +292,20 @@ The pre-defined Untrusted Non3pp Access flavor provisions the minimal set of con The following are steps to install this flavor: -1. Configure the OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). - -2. Run the x-OEK deployment script: - - ```bash - $ ido-openness-experience-kits# deploy_ne.sh -f untrusted-non3pp-access - ``` +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `untrusted-non3pp-access` + ```yaml + --- + all: + vars: + cluster_name: untrusted_non3pp_access_cluster + flavor: untrusted-non3pp-access + ... + ``` +3. Run ido-CEEK deployment script: + ```shell + $ python3 deploy.py + ``` This deployment flavor enables the following ingredients: @@ -231,12 +320,21 @@ This deployment flavor enables the following ingredients: The pre-defined CERA Near Edge flavor provisions the required set of configurations for a 5G Converged Edge Reference Architecture for Near Edge deployments on Intel® Xeon® platforms. The following are steps to install this flavor: -1. Configure the OEK under CERA repository as described in the [Converged Edge Reference Architecture Near Edge](https://github.com/open-ness/ido-specs/blob/master/doc/reference-architectures/CERA-Near-Edge.md). - -2. Run the x-OEK for CERA deployment script: - ```shell - $ ido-converged-edge-experience-kits# deploy_openness_for_cera.sh - ``` +1. Configure the CEEK under CERA repository as described in the [Converged Edge Reference Architecture Near Edge](https://github.com/open-ness/ido-specs/blob/master/doc/reference-architectures/CERA-Near-Edge.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `cera_5g_near_edge` + ```yaml + --- + all: + vars: + cluster_name: cera_5g_near_edge_cluster + flavor: cera_5g_near_edge + single_node_deployment: true + ... + ``` +3. Run ido-CEEK deployment script: + ```shell + $ python3 deploy.py + ``` This deployment flavor enables the following ingredients: @@ -255,12 +353,21 @@ This deployment flavor enables the following ingredients: The pre-defined CERA Near Edge flavor provisions the required set of configurations for a 5G Converged Edge Reference Architecture for On Premises deployments on Intel® Xeon® platforms. It also provisions for deployment of Intel® FPGA Programmable Acceleration Card (Intel® FPGA PAC) N3000 tools and components to enable offloading for the acceleration of FEC (Forward Error Correction) to the FPGA. The following are steps to install this flavor: -1. Configure the OEK under CERA repository as described in the [Converged Edge Reference Architecture On Premises Edge](https://github.com/open-ness/ido-specs/blob/master/doc/reference-architectures/CERA-5G-On-Prem.md). - -2. Run the x-OEK for CERA deployment script: - ```shell - $ ido-converged-edge-experience-kits# deploy_openness_for_cera.sh - ``` +1. Configure the CEEK under CERA repository as described in the [Converged Edge Reference Architecture On Premises Edge](https://github.com/open-ness/ido-specs/blob/master/doc/reference-architectures/CERA-5G-On-Prem.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `cera_5g_on_premise` + ```yaml + --- + all: + vars: + cluster_name: cera_5g_on_premise_cluster + flavor: cera_5g_on_premise + single_node_deployment: true + ... + ``` +3. Run ido-CEEK deployment script: + ```shell + $ python3 deploy.py + ``` This deployment flavor enables the following ingredients: @@ -277,43 +384,35 @@ This deployment flavor enables the following ingredients: - HugePages of size 1Gi and the amount of HugePages as 40G for the nodes - RMD operator -## Reference Service Mesh - -Service Mesh technology enables services discovery and sharing of data between application services. This technology can be useful in any CERA. Customers will find Service Mesh under flavors directory as a reference to quickly try out the technology and understand the implications. In future OpenNESS releases this Service Mesh will not be a dedicated flavor. - -The pre-defined *service-mesh* deployment flavor installs the OpenNESS service mesh that is based on [Istio](https://istio.io/). +## CERA 5G Central Office Flavor -> **NOTE**: When deploying Istio Service Mesh in VMs, a minimum of 8 CPU core and 16GB RAM must be allocated to each worker VM so that Istio operates smoothly +The pre-defined CERA 5g Central Office flavor provisions the required set of configurations for a 5G Converged Edge Reference Architecture for Core Network applications deployments on Intel® Xeon® platforms. -Steps to install this flavor are as follows: -1. Configure OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run OEK deployment script: +The following are steps to install this flavor: +1. Configure the CEEK under CERA repository as described in the [Converged Edge Reference Architecture On Premises Edge](reference-architectures/CERA-5G-On-Prem.md) or [Converged Edge Reference Architecture Near Edge](reference-architectures/CERA-Near-Edge.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `cera_5g_central_office` + ```yaml + --- + all: + vars: + cluster_name: cera_5g_central_office_cluster + flavor: cera_5g_central_office + single_node_deployment: true + ... + ``` +3. Run ido-CEEK deployment script: ```shell - $ deploy_ne.sh -f service-mesh + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: -* Node Feature Discovery -* The default Kubernetes CNI: `kube-ovn` -* Istio service mesh -* Kiali management console -* Telemetry - -> **NOTE:** Kiali management console username can be changed by editing the variable `istio_kiali_username`. By default `istio_kiali_password` is randomly generated and can be retirieved by running `kubectl get secrets/kiali -n istio-system -o json | jq -r '.data.passphrase' | base64 -d` on the Kubernetes controller. - -Following parameters in the flavor/all.yaml can be customize for Istio deployment: - -```code -# Istio deployment profile possible values: default, demo, minimal, remote -istio_deployment_profile: "default" -# Kiali -istio_kiali_username: "admin" -istio_kiali_password: "{{ lookup('password', '/dev/null length=16') }}" -istio_kiali_nodeport: 30001 -``` - -> **NOTE:** If creating a customized flavor, the Istio service mesh installation can be included in the Ansible playbook by setting the flag `ne_istio_enable: true` in the flavor file. +- Kubernetes CNI: Calico and SRIOV. +- SRIOV device plugin +- Virtual Functions +- Kubernetes Device Plugin +- BIOSFW feature +- HugePages of size 8Gi and the amount of HugePages as 40G for the nodes ## Central Orchestrator Flavor @@ -321,14 +420,95 @@ Central Orchestrator Flavor is used to deploy EMCO. The pre-defined *orchestration* deployment flavor provisions an optimized system configuration for emco (central orchestrator) workloads on Intel Xeon servers. It also provisions a set of central orchestrator services for [edge, multiple clusters orchestration](building-blocks/emco/openness-emco.md). -Steps to install this flavor are as follows: -1. Configure OEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/network-edge/controller-edge-node-setup.md). -2. Run OEK deployment script: +The following are steps to install this flavor: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Update the `inventory.yaml` file by setting the deployment flavor as `central_orchestrator` + ```yaml + --- + all: + vars: + cluster_name: central_orchestrator_cluster + flavor: central_orchestrator + ... + ``` +3. Run CEEK deployment script: ```shell - $ deploy_ne.sh -f central_orchestrator + $ python3 deploy.py ``` This deployment flavor enables the following ingredients: * Harbor Registry -* The default Kubernetes CNI: `kube-ovn` -* EMCO services \ No newline at end of file +* The default Kubernetes CNI: `calico` +* EMCO services + +## CERA SD-WAN Edge Flavor + +CERA SD-WAN Edge flavor is used to deploy SD-WAN on the OpenNESS cluster acting as an Edge platform. This CERA flavor only supports single-node OpenNESS deployments. It provides configuration that supports running SD-WAN CNFs on the OpenNESS cluster, enables hardware accelerators with the HDDL plugin, and adds support for service mesh and node feature disovery to aid other applications and services runing on the Edge node. This CERA flavor disbless EAA, Kafka adn Edge DNS services for platform optimization. + +The following are steps to install this flavor: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Configure the CNF as described in [Converged Edge Reference Architecture for SD-WAN](reference-architectures/cera_sdwan.md#ewo-configuration). +3. Update the `inventory.yaml` file by setting the deployment flavor as `sdewan-edge` + ```yaml + --- + all: + vars: + cluster_name: sdewan_edge_cluster + flavor: sdewan-edge + single_node_deployment: true + ... + ``` +4. Run CEEK deployment script: + ```shell + $ python3 deploy.py + ``` + +This CERA flavor enables the following deployment configuration: +* Istio servise mesh on the default namespace +* Node Feature Discovery +* The primamary K8s CNI: 'calico' +* The secondary K8s CNI: 'ovn4nfv' +* HDDL support +* Telemetry +* Reserved CPUs for K8s and OS daemons +* Kiali management console + +This CERA flavor disables the following deployment configuration: +* EAA service with Kafka +* Edge DNS + +## CERA SD-WAN Hub Flavor + +CERA SD-WAN Hub flavor is used to deploy SD-WAN on the OpenNESS cluster acting as a Hub for Edge clusters. It only supports single-node OpenNESS deployments. This CERA flavor disabless EAA, Kafka and EAA services for platform optimization. + +The following are steps to install this flavor: +1. Configure the CEEK as described in the [OpenNESS Getting Started Guide for Network Edge](getting-started/openness-cluster-setup.md). +2. Configure the CNF as described in [Converged Edge Reference Architecture for SD-WAN](reference-architectures/cera_sdwan.md#ewo-configuration). +3. Update the `inventory.yaml` file by setting the deployment flavor as `sdewan-hub` + ```yaml + --- + all: + vars: + cluster_name: sdewan_hub_cluster + flavor: sdewan-hub + single_node_deployment: true + ... + ``` +4. Run CEEK deployment script: + ```shell + $ python3 deploy.py + ``` + +This CERA flavor enables the following deployment configuration: +* The primamary CNI 'calico' +* The secondary CNI 'ovn4nfv' +* Telemetry +* Reserved CPUs for K8s and OS daemons +* Kiali management console + + +This CERA flavor disables the following deployemnt configuration: +* Node Feature Discovery +* EAA service with Kafka +* Edge DNS +* HDDL support diff --git a/doc/getting-started/converged-edge-experience-kits.md b/doc/getting-started/converged-edge-experience-kits.md new file mode 100644 index 00000000..d6db91b0 --- /dev/null +++ b/doc/getting-started/converged-edge-experience-kits.md @@ -0,0 +1,416 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2021 Intel Corporation +``` + +# Converged Edge Experience Kits +- [Purpose](#purpose) +- [Converged Edge Experience Kit explained](#converged-edge-experience-kit-explained) +- [The inventory file](#the-inventory-file) +- [Sample Deployment Definitions](#sample-deployment-definitions) + - [Single Cluster Deployment](#single-cluster-deployment) + - [Single-node Cluster Deployment](#single-node-cluster-deployment) + - [Multi-cluster deployment](#multi-cluster-deployment) +- [Deployment customization](#deployment-customization) +- [Customizing kernel, grub parameters, and tuned profile & variables per host](#customizing-kernel-grub-parameters-and-tuned-profile--variables-per-host) + - [IP address range allocation for various CNIs and interfaces](#ip-address-range-allocation-for-various-cnis-and-interfaces) + - [Default values](#default-values) + - [Use different realtime kernel (3.10.0-1062)](#use-different-realtime-kernel-3100-1062) + - [Use different non-rt kernel (3.10.0-1062)](#use-different-non-rt-kernel-3100-1062) + - [Use tuned 2.9](#use-tuned-29) + - [Default kernel and configure tuned](#default-kernel-and-configure-tuned) + - [Change amount of HugePages](#change-amount-of-hugepages) + - [Change size of HugePages](#change-size-of-hugepages) + - [Change amount and size of HugePages](#change-amount-and-size-of-hugepages) + - [Remove input output memory management unit (IOMMU) from grub params](#remove-input-output-memory-management-unit-iommu-from-grub-params) + - [Add custom GRUB parameter](#add-custom-grub-parameter) + - [Configure OVS-DPDK in kube-ovn](#configure-ovs-dpdk-in-kube-ovn) +- [Adding new CNI plugins for Kubernetes (Network Edge)](#adding-new-cni-plugins-for-kubernetes-network-edge) + +## Purpose + +The Converged Edge Experience Kit is a refreshed repository of Ansible\* playbooks for automated deployment of Converged Edge Reference Architectures. + +The Converged Edge Experience Kit introduces the following capabilities: +1. Wide range of deployments from individual building blocks to full end-to-end reference deployments +2. Minimal to near-zero user interventions.. Typically, the user provides the details of the nodes that constitute the OpenNESS edge cluster and executes the deployment script +3. More advanced deployments can be customized in the form of Ansible\* group and host variables. This mode requires users with in-depth knowledge and expertise of the subject edge deployment +4. Enablement of end-to-end multi-cluster deployments such as Near Edge and On-premises reference architectures + +## Converged Edge Experience Kit explained +The Converged Edge Experience Kit repository is organized as detailed in the following structure: +``` +├── cloud +├── flavors +├── inventory +│ ├── automated +│ └── default +│ ├── group_vars +│ │ └── all +│ │ └── 10-default.yml +│ └── host_vars +├── playbooks +│ ├── infrastructure.yml +│ ├── kubernetes.yml +│ └── applications.yml +├── roles +│ ├── applications +│ ├── infrastructure +│ ├── kubernetes +│ └── telemetry +├── scripts +├── tasks +├── inventory.yml +├── network_edge_cleanup.yml +└── deploy.py +``` + +* `flavors`: definition variables of pre-defined deployment flavors +* `inventory`: definition of default & generated Ansible\* variables +* `inventory/default/group_vars/all/10-default.yml`: definition of default variables for all deployments +* `inventory/automated`: inventory files that were automatically generated by the deployment helper script +* `playbooks`: Ansible\* playbooks for infrastructure, Kubernetes and applications +* `roles`: Ansible roles for infrastructure, Kubernetes, applications and telemetry +* `scripts`: utility scripts +* `inventory.yml`: definition of the clusters, their controller & edge nodes and respective deployment flavors +* `deploy.py`: the deployment helper script + + +## The inventory file +The inventory file defines the group of physical nodes that constitute the edge cluster which will be deployed by the Converged Edge Experience Kits. The inventory file YAML specification allows deploying multiple edge clusters in one command run. Multiple clusters must be separated by the 3 dashes `---` directive. + +> **NOTE**: for multi-cluster deployments, user must assign distinct names to the controller and the edge nodes, i.e., no hostname repetitions. + +The following variables must be defined + +* `cluster_name`: a given name for the OpenNESS edge cluster deployment - separated by underscores `_` instead of spaces. +* `flavor`: the deployment flavor applicable for the OpenNESS edge deployment as defined in the [Deployment flavors](../flavors.md) document. +* `single_node_deployment`: If set to `true`, a single-node cluster is deployed.. Must satisfy the following conditions: + - IP address (`ansible_host`) for both controller and node must be the same + - `controller_group` and `edgenode_group` groups must contain exactly one host +* `limit` -- **OPTIONAL**: constrains the deployment to a specific Ansible\* group, e.g., `controller`, `edgenode`, `edgenode_vca_group` or just a particular hostname. This is passed as a `--limit` command-line option when executing `ansible-playbook`. + +## Sample Deployment Definitions +### Single Cluster Deployment +Set `single_node_deployment` flag to `false` in the inventory file and provide the controller node name under the `controller_group` and the edge node names under the `edgenode_group`. + +Example: + +```yaml +--- +all: + vars: + cluster_name: 5g_near_edge + flavor: cera_5g_near_edge + single_node_deployment: false + limit: +controller_group: + hosts: + ctrl.openness.org: + ansible_host: 10.102.227.154 + ansible_user: openness +edgenode_group: + hosts: + node01.openness.org: + ansible_host: 10.102.227.11 + ansible_user: openness + node02.openness.org: + ansible_host: 10.102.227.79 + ansible_user: openness +edgenode_vca_group: + hosts: +ptp_master: + hosts: +ptp_slave_group: + hosts: +``` + +### Single-node Cluster Deployment +Set `single_node_deployment` flag to `true` in the inventory file and provide the node name in the `controller_group` and the `edgenode_group`. + +Example: + +```yaml +--- +all: + vars: + cluster_name: 5g_central_office + flavor: cera_5g_central_office + single_node_deployment: true + limit: +controller_group: + hosts: + node.openness.org: + ansible_host: 10.102.227.234 + ansible_user: openness +edgenode_group: + hosts: + node.openness.org: + ansible_host: 10.102.227.234 + ansible_user: openness +edgenode_vca_group: + hosts: +ptp_master: + hosts: +ptp_slave_group: + hosts: +``` + +### Multi-cluster deployment +Provide multiple clusters YAML specifications separated by the 3 dashes `---` directive in the inventory.yml. A node name should be used only once across the inventory file, i.e: distinct node names. + +Example: + +```yaml +--- +all: + vars: + cluster_name: 5g_near_edge + flavor: cera_5g_near_edge + single_node_deployment: true + limit: +controller_group: + hosts: + node.openness01.org: + ansible_host: 10.102.227.154 + ansible_user: openness +edgenode_group: + hosts: + node.openness01.org: + ansible_host: 10.102.227.154 + ansible_user: openness +edgenode_vca_group: + hosts: +ptp_master: + hosts: +ptp_slave_group: + hosts: +--- +all: + vars: + cluster_name: 5g_central_office + flavor: cera_5g_central_office + single_node_deployment: true + limit: +controller_group: + hosts: + node.openness02.org: + ansible_host: 10.102.227.234 + ansible_user: openness +edgenode_group: + hosts: + node.openness02.org: + ansible_host: 10.102.227.234 + ansible_user: openness +edgenode_vca_group: + hosts: +ptp_master: + hosts: +ptp_slave_group: + hosts: +``` + +## Deployment customization +The `deploy.py` script creates a new inventory for each cluster to be deployed in a `inventory/automated` directory. These inventories are based on `inventory/default` - all directories and files are symlinked. Additionally, relevant flavor files are symlinked. + +Customizations made to `inventory/default/group_vars` and `inventory/default/host_vars` will affect every deployment performed by `deploy.py` (because these files are symlinked, not copied). Therefore it is a good place to provide changes relevant to the nodes of the cluster. + +## Customizing kernel, grub parameters, and tuned profile & variables per host + +CEEKs allow a user to customize kernel, grub parameters, and tuned profiles by leveraging Ansible's feature of `host_vars`. + +> **NOTE**: `inventory/default/groups_vars/[edgenode|controller|edgenode_vca]_group` directories contain variables applicable for the respective groups and they can be used in `inventory/default/host_vars` to change on per node basis while `inventory/default/group_vars/all` contains cluster wide variables. + +CEEKs contain a `inventory/default/host_vars/` directory in which we can create another directory (`nodes-inventory-name`) and place a YAML file (`10-open.yml`, e.g., `node01/10-open.yml`). The file would contain variables that would override roles' default values. + +> **NOTE**: Despite the ability to customize parameters (kernel), it is required to have a clean CentOS\* 7.9.2009 operating system installed on hosts (from a minimal ISO image) that will be later deployed from Ansible scripts. This OS shall not have any user customizations. + +To override the default value, place the variable's name and new value in the host's vars file. For example, the contents of `inventory/default/host_vars/node01/10-open.yml` that would result in skipping kernel customization on that node: + +```yaml +kernel_skip: true +``` + +The following are several common customization scenarios. + +### IP address range allocation for various CNIs and interfaces + +The Converged Edge Experience kits deployment uses/allocates/reserves a set of IP address ranges for different CNIs and interfaces. The server or host IP address should not conflict with the default address allocation. +In case if there is a critical need for the server IP address used by the OpenNESS default deployment, it would require to modify the default addresses used by the OpenNESS. + +Following files specify the CIDR for CNIs and interfaces. These are the IP address ranges allocated and used by default just for reference. + +```yaml +flavors/media-analytics-vca/all.yml:19:vca_cidr: "172.32.1.0/12" +inventory/default/group_vars/all/10-open.yml:90:calico_cidr: "10.245.0.0/16" +inventory/default/group_vars/all/10-open.yml:93:flannel_cidr: "10.244.0.0/16" +inventory/default/group_vars/all/10-open.yml:96:weavenet_cidr: "10.32.0.0/12" +inventory/default/group_vars/all/10-open.yml:99:kubeovn_cidr: "10.16.0.0/16,100.64.0.0/16,10.96.0.0/12" +roles/kubernetes/cni/kubeovn/controlplane/templates/crd_local.yml.j2:13: cidrBlock: "192.168.{{ loop.index0 + 1 }}.0/24" +``` + +The `192.168.*.*` is used for SRIOV and interface service IP address allocation in Kube-ovn CNI. So it is not allowed for the server IP address, which conflicting with this range. +Completely avoid the range of address defined as per the netmask as it may conflict in routing rules. + +E.g., If the server/host IP address is required to use `192.168.*.*` while this range by default used for SRIOV interfaces in OpenNESS. The IP address range for `cidrBlock` in `roles/kubernetes/cni/kubeovn/controlplane/templates/crd_local.yml.j2` file can be changed to `192.167.{{ loop.index0 + 1 }}.0/24` to use some other IP segment for SRIOV interfaces. + + +### Default values +Here are several default values: + +```yaml +# --- machine_setup/custom_kernel +kernel_skip: false # use this variable to disable custom kernel installation for host + +kernel_repo_url: http://linuxsoft.cern.ch/cern/centos/7.9.2009/rt/CentOS-RT.repo +kernel_repo_key: http://linuxsoft.cern.ch/cern/centos/7.9.2009/os/x86_64/RPM-GPG-KEY-cern +kernel_package: kernel-rt-kvm +kernel_devel_package: kernel-rt-devel +kernel_version: 3.10.0-1160.11.1.rt56.1145.el7.x86_64 + +kernel_dependencies_urls: [] +kernel_dependencies_packages: [] + +# --- machine_setup/grub +hugepage_size: "2M" # Or 1G +hugepage_amount: "5000" + +default_grub_params: "hugepagesz={{ hugepage_size }} hugepages={{ hugepage_amount }} intel_iommu=on iommu=pt" +additional_grub_params: "" + +# --- machine_setup/configure_tuned +tuned_skip: false # use this variable to skip tuned profile configuration for host +tuned_packages: + - tuned-2.11.0-9.el7 + - http://ftp.scientificlinux.org/linux/scientific/7/x86_64/os/Packages/tuned-profiles-realtime-2.11.0-9.el7.noarch.rpm +tuned_profile: realtime +tuned_vars: | + isolated_cores=2-3 + nohz=on + nohz_full=2-3 +``` + +### Use different realtime kernel (3.10.0-1062) +By default, `kernel-rt-kvm-3.10.0-1160.11.1.rt56.1145.el7.x86_64` from the built-in repository is installed. + +To use another version (e.g., `kernel-rt-kvm-3.10.0-1062.9.1.rt56.1033.el7.x86_64`), create a `host_var` file for the host with content: +```yaml +kernel_version: 3.10.0-1062.9.1.rt56.1033.el7.x86_64 +``` + +### Use different non-rt kernel (3.10.0-1062) +The CEEK installs a real-time kernel by default. However, the non-rt kernel is present in the official CentOS repository. Therefore, to use a different non-rt kernel, the following overrides must be applied: +```yaml +kernel_repo_url: "" # package is in default repository, no need to add new repository +kernel_package: kernel # instead of kernel-rt-kvm +kernel_devel_package: kernel-devel # instead of kernel-rt-devel +kernel_version: 3.10.0-1062.el7.x86_64 + +dpdk_kernel_devel: "" # kernel-devel is in the repository, no need for url with RPM + +# Since, we're not using rt kernel, we don't need a tuned-profiles-realtime but want to keep the tuned 2.11 +tuned_packages: +- http://linuxsoft.cern.ch/scientific/7x/x86_64/os/Packages/tuned-2.11.0-8.el7.noarch.rpm +tuned_profile: balanced +tuned_vars: "" +``` + +### Use tuned 2.9 +```yaml +tuned_packages: +- tuned-2.9.0-1.el7fdp +- tuned-profiles-realtime-2.9.0-1.el7fdp +``` + +### Default kernel and configure tuned +```yaml +kernel_skip: true # skip kernel customization altogether + +# update tuned to 2.11 but don't install tuned-profiles-realtime since we're not using rt kernel +tuned_packages: +- http://linuxsoft.cern.ch/scientific/7x/x86_64/os/Packages/tuned-2.11.0-8.el7.noarch.rpm +tuned_profile: balanced +tuned_vars: "" +``` + +### Change amount of HugePages +```yaml +hugepage_amount: "1000" # default is 5000 +``` + +### Change size of HugePages +```yaml +hugepage_size: "1G" # default is 2M +``` + +### Change amount and size of HugePages +```yaml +hugepage_amount: "10" # default is 5000 +hugepage_size: "1G" # default is 2M +``` + +### Remove input output memory management unit (IOMMU) from grub params +```yaml +default_grub_params: "hugepagesz={{ hugepage_size }} hugepages={{ hugepage_amount }}" +``` + +### Add custom GRUB parameter +```yaml +additional_grub_params: "debug" +``` + +### Configure OVS-DPDK in kube-ovn +By default, OVS-DPDK is disabled (due to set calico as a default cni). To enable it, set a flag: +```yaml +kubeovn_dpdk: true +``` + +> **NOTE**: This flag should be set in `roles/kubernetes/cni/kubeovn/common/defaults/main.ym` or added to `inventory/default/group_vars/all/10-open.yml`. + +Additionally, HugePages in the OVS pod can be adjusted once default HugePage settings are changed. +```yaml +kubeovn_dpdk_socket_mem: "1024,0" # Amount of hugepages reserved for OVS per NUMA node (node 0, node 1, ...) in MB +kubeovn_dpdk_hugepage_size: "2Mi" # Default size of hugepages, can be 2Mi or 1Gi +kubeovn_dpdk_hugepages: "1Gi" # Total amount of hugepages that can be used by OVS-OVN pod +``` + +> **NOTE**: If the machine has multiple NUMA nodes, remember that HugePages must be allocated for **each NUMA node**. For example, if a machine has two NUMA nodes, `kubeovn_dpdk_socket_mem: "1024,1024"` or similar should be specified. + +>**NOTE**: If `kubeovn_dpdk_socket_mem` is changed, set the value of `kubeovn_dpdk_hugepages` to be equal to or greater than the sum of `kubeovn_dpdk_socket_mem` values. For example, for `kubeovn_dpdk_socket_mem: "1024,1024"`, set `kubeovn_dpdk_hugepages` to at least `2Gi` (equal to 2048 MB). + +>**NOTE**: `kubeovn_dpdk_socket_mem`, `kubeovn_dpdk_pmd_cpu_mask`, and `kubeovn_dpdk_lcore_mask` can be set on per node basis but the HugePage amount allocated with `kubeovn_dpdk_socket_mem` cannot be greater than `kubeovn_dpdk_hugepages`, which is the same for the whole cluster. + +OVS pods limits are configured by: +```yaml +kubeovn_dpdk_resources_requests: "1Gi" # OVS-OVN pod RAM memory (requested) +kubeovn_dpdk_resources_limits: "1Gi" # OVS-OVN pod RAM memory (limit) +``` +CPU settings can be configured using: +```yaml +kubeovn_dpdk_pmd_cpu_mask: "0x4" # DPDK PMD CPU mask +kubeovn_dpdk_lcore_mask: "0x2" # DPDK lcore mask +``` + +## Adding new CNI plugins for Kubernetes (Network Edge) + +* The role that handles CNI deployment must be placed in the `roles/kubernetes/cni/` directory (e.g., `roles/kubernetes/cni/kube-ovn/`). +* Subroles for control plane and node (if needed) should be placed in the `controlplane/` and `node/` directories (e.g., `roles/kubernetes/cni/kube-ovn/{controlplane,node}`). +* If there is a part of common command for both control plane and node, additional sub-roles can be created: `common` (e.g., `roles/kubernetes/cni/sriov/common`).+>**NOTE**: The automatic inclusion of the `common` role should be handled by Ansible mechanisms (e.g., usage of meta's `dependencies` or `include_role` module) +* Name of the main role must be added to the `available_kubernetes_cnis` variable in `roles/kubernetes/cni/defaults/main.yml`. +* If additional requirements must checked before running the playbook (to not have errors during execution), they can be placed in the `roles/kubernetes/cni/tasks/precheck.yml` file, which is included as a pre_task in plays for both Edge Controller and Edge Node.
+The following are basic prechecks that are currently executed: + * Check if any CNI is requested (i.e., `kubernetes_cni` is not empty). + * Check if `sriov` is not requested as primary (first on the list) or standalone (only on the list). + * Check if `calico` is requested as a primary (first on the list). + * Check if `kubeovn` is requested as a primary (first on the list). + * Check if the requested CNI is available (check if some CNI is requested that isn't present in the `available_kubernetes_cnis` list). +* CNI roles should be as self-contained as possible (unless necessary, CNI-specific tasks should not be present in `kubernetes/{controlplane,node,common}` or `openness/network_edge/{controlplane,node}`). +* If the CNI needs a custom OpenNESS service (e.g., Interface Service in case of `kube-ovn`), it can be added to the `openness/network_edge/{controlplane,node}`.
+ Preferably, such tasks would be contained in a separate task file (e.g., `roles/openness/controlplane/tasks/kube-ovn.yml`) and executed only if the CNI is requested. For example: + ```yaml + - name: deploy interface service for kube-ovn + include_tasks: kube-ovn.yml + when: "'kubeovn' in kubernetes_cnis" + ``` +* If the CNI is used as an additional CNI (with Multus\*), the network attachment definition must be supplied ([refer to Multus docs for more info](https://github.com/intel/multus-cni/blob/master/docs/quickstart.md#storing-a-configuration-as-a-custom-resource)). diff --git a/doc/getting-started/harbor-registry.md b/doc/getting-started/harbor-registry.md new file mode 100644 index 00000000..11469d84 --- /dev/null +++ b/doc/getting-started/harbor-registry.md @@ -0,0 +1,206 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2021 Intel Corporation +``` + +# Harbor Registry Service in OpenNESS +- [Deploy Harbor registry](#deploy-harbor-registry) + - [System Prerequisite](#system-prerequisite) + - [Ansible Playbooks](#ansible-playbooks) + - [Projects](#projects) +- [Harbor login](#harbor-login) +- [Harbor registry image push](#harbor-registry-image-push) +- [Harbor registry image pull](#harbor-registry-image-pull) +- [Harbor UI](#harbor-ui) +- [Harbor CLI](#harbor-cli) + - [CLI - List Project](#cli---list-project) + - [CLI - List Image Repositories](#cli---list-image-repositories) + - [CLI - Delete Image](#cli---delete-image) + +Harbor registry is an open source cloud native registry which can support images and relevant artifacts with extended functionalities as described in [Harbor](https://goharbor.io/). On the OpenNESS environment, Harbor registry service is installed on Control plane Node by Harbor Helm Chart [github](https://github.com/goharbor/harbor-helm/releases/tag/v1.5.1). Harbor registry authentication enabled with self-signed certificates as well as all nodes and control plane will have access to the Harbor registry. + +## Deploy Harbor registry + +### System Prerequisite +* The available system disk should be reserved at least 20G for Harbor PV/PVC usage. The defaut disk PV/PVC total size is 20G. The values can be configured in the ```roles/harbor_registry/controlplane/defaults/main.yaml```. +* If huge pages enabled, need 1G(hugepage size 1G) or 300M(hugepage size 2M) to be reserved for Harbor usage. + +### Ansible Playbooks +Ansible `harbor_registry` roles created in Converged Edge Experience Kits. For deploying a Harbor registry on Kubernetes, control plane roles are enabled in the main `network_edge.yml` playbook file. + +```ini +role: harbor_registry/controlplane +role: harbor_registry/node +``` + +The following steps are processed by converged-edge-experience-kits during the Harbor registry installation on the OpenNESS control plane node. + +* Download Harbor Helm Charts on the Kubernetes Control plane Node. +* Check whether huge pages is enabled and templates values.yaml file accordingly. +* Create namespace and disk PV for Harbor Services (The default disk PV/PVC total size is 20G. The values can be configured in the `roles/kubernetes/harbor_registry/controlplane/defaults/main.yaml`). +* Install Harbor on the control plane node using the Helm Charts (The CA crt will be generated by Harbor itself). +* Create the new project - ```intel``` for OpenNESS microservices, Kurbernetes enhanced add-on images storage. +* Docker login the Harbor Registry, thus enable pulling, pushing and tag images with the Harbor Registry + + +On the OpenNESS edge nodes, converged-edge-experience-kits will conduct the following steps: +* Get harbor.crt from the OpenNESS control plane node and save into the host location + /etc/docker/certs.d/
You are being redirected to the OpenNESS Docs.
diff --git a/doc/building-blocks/enhanced-platform-awareness/openness-kubernetes-dashboard.md b/doc/getting-started/kubernetes-dashboard.md similarity index 80% rename from doc/building-blocks/enhanced-platform-awareness/openness-kubernetes-dashboard.md rename to doc/getting-started/kubernetes-dashboard.md index 56482394..17a2e94a 100644 --- a/doc/building-blocks/enhanced-platform-awareness/openness-kubernetes-dashboard.md +++ b/doc/getting-started/kubernetes-dashboard.md @@ -17,7 +17,7 @@ Kubernetes Dashboard is a web user interface for Kubernetes. User can use Dashbo ## Details - Kubernetes Dashboard support in OpenNESS -Kubernetes Dashboard is disabled by default in OpenNESS Experience Kits. It can be enabled by setting variable `kubernetes_dashboard_enable` in `group_vars/all/10-default.yml` file to `true` value: +Kubernetes Dashboard is disabled by default in Converged Edge Experience Kits. It can be enabled by setting variable `kubernetes_dashboard_enable` in `inventory/default/group_vars/all/10-open.yml` file to `true` value: ```yaml # Kubernetes Dashboard @@ -26,7 +26,7 @@ kubernetes_dashboard_enable: false # set to true to enable Kubernetes Dashboard ### TLS encryption -TLS for Kubernetes dashboard is enabled by default. User can disable TLS encryption using variable `disable_dashboard_tls` in `group_vars/all/10-default.yml`: +TLS for Kubernetes dashboard is enabled by default. User can disable TLS encryption using variable `disable_dashboard_tls` in `inventory/default/group_vars/all/10-open.yml`: ```yaml disable_dashboard_tls: false # set to true to disable TLS @@ -34,7 +34,7 @@ disable_dashboard_tls: false # set to true to disable TLS ### Usage -User can use Kubernetes Dashboard by browsing `https://-To deploy Network Edge in a single-node cluster scenario, follow the steps below: -1. Modify `inventory.ini`
- > Rules for inventory: - > - IP address (`ansible_host`) for both controller and node must be the same - > - `edgenode_group` and `controller_group` groups must contain exactly one host - - Example of a valid inventory: - ```ini - [all] - controller ansible_ssh_user=root ansible_host=192.168.0.11 - node01 ansible_ssh_user=root ansible_host=192.168.0.11 - - [controller_group] - controller - - [edgenode_group] - node01 - - [edgenode_vca_group] - ``` -2. Features can be enabled in the `group_vars/all/10-open.yml` file by tweaking the configuration variables. -3. Settings regarding the kernel, grub, HugePages\*, and tuned can be customized in `group_vars/edgenode_group/10-open.yml`. - > Default settings in the single-node cluster mode are those of the Edge Node (i.e., kernel and tuned customization enabled). -4. Single-node cluster can be deployed by running command: `./deploy_ne.sh single` - -## Harbor registry - -Harbor registry is an open source cloud native registry which can support images and relevant artifacts with extended functionalities as described in [Harbor](https://goharbor.io/). On the OpenNESS environment, Harbor registry service is installed on Control plane Node by Harbor Helm Chart [github](https://github.com/goharbor/harbor-helm/releases/tag/v1.5.1). Harbor registry authentication enabled with self-signed certificates as well as all nodes and control plane will have access to the Harbor registry. - -### Deploy Harbor registry - -#### System Prerequisite -* The available system disk should be reserved at least 20G for Harbor PV/PVC usage. The defaut disk PV/PVC total size is 20G. The values can be configurable in the ```roles/harbor_registry/controlplane/defaults/main.yaml```. -* If huge pages enabled, need 1G(hugepage size 1G) or 300M(hugepage size 2M) to be reserved for Harbor usage. - -#### Ansible Playbooks -Ansible "harbor_registry" roles created on openness-experience-kits. For deploying a Harbor registry on Kubernetes, control plane roles are enabled on the openness-experience-kits "network_edge.yml" file. - - ```ini - role: harbor_registry/controlplane - role: harbor_registry/node - ``` - -The following steps are processed by openness-experience-kits during the Harbor registry installation on the OpenNESS control plane node. - -* Download Harbor Helm Charts on the Kubernetes Control plane Node. -* Check whether huge pages is enabled and templates values.yaml file accordingly. -* Create namespace and disk PV for Harbor Services (The defaut disk PV/PVC total size is 20G. The values can be configurable in the ```roles/harbor_registry/controlplane/defaults/main.yaml```). -* Install Harbor on the control plane node using the Helm Charts (The CA crt will be generated by Harbor itself). -* Create the new project - ```intel``` for OpenNESS microservices, Kurbernetes enhanced add-on images storage. -* Docker login the Harbor Registry, thus enable pulling, pushing and tag images with the Harbor Registry - - -On the OpenNESS edge nodes, openness-experience-kits will conduct the following steps: -* Get harbor.crt from the OpenNESS control plane node and save into the host location - /etc/docker/certs.d/
You are being redirected to the OpenNESS Docs.
- diff --git a/doc/getting-started/network-edge/offline-edge-deployment.md b/doc/getting-started/network-edge/offline-edge-deployment.md deleted file mode 100644 index 0b1fb4b3..00000000 --- a/doc/getting-started/network-edge/offline-edge-deployment.md +++ /dev/null @@ -1,157 +0,0 @@ -```text -SPDX-License-Identifier: Apache-2.0 -Copyright (c) 2019-2020 Intel Corporation -``` - -- [OpenNESS Network Edge: Offline Deployment](#openness-network-edge-offline-deployment) - - [OpenNESS support in offline environment](#openness-support-in-offline-environment) - - [Setup prerequisites](#setup-prerequisites) - - [Creating the offline package from an online node](#creating-the-offline-package-from-an-online-node) - - [Placing the complete offline package in offline environment](#placing-the-complete-offline-package-in-offline-environment) - - [Deployment in offline environment](#deployment-in-offline-environment) -# OpenNESS Network Edge: Offline Deployment - -## OpenNESS support in offline environment - -The OpenNESS projects supports a deployment of the solution in an air-gapped, offline environment. The support is currently limited to "[flexran" deployment flavor of OpenNESS Experience Kit](https://github.com/open-ness/ido-openness-experience-kits/tree/master/flavors/flexran) only and it allows for offline deployment of vRAN specific components. Internet connection is needed to create the offline package, a script to download and build all necessary components will create an archive of all the necessary files. Once the offline package is created the installation of OpenNESS Experience Kits will be commenced as usual, in the same way as the default online installation would. - -It can be deployed in two different scenarios. The first scenario is to deploy the OpenNESS Experience Kits from the online "jumper" node which is being used to create the offline package, this internet connected node must have a network connection to the air-gapped/offline nodes. The second scenario is to copy the whole OpenNESS Experience Kit directory with the already archived packages to the air-gapped/offline environment (for example via USB or other media or means) and run the OpenNESS Experience Kit from within the offline environment. All the nodes within the air-gapped/offline cluster need to able to SSH into each other. - -Figure 1. Scenario one - online node connected to the air-gapped network - -Figure 2. Scenario two - OEK copied to the air-gapped network - - -## Setup prerequisites - -* A node with access to internet to create the offline package. -* Cluster set up in an air-gapped environment. -* Clean setup, see [pre-requisites](https://github.com/open-ness/ido-specs/blob/master/doc/getting-started/network-edge/controller-edge-node-setup.md#preconditions) -* [Optional] If OEK is run from an online jumper node, the node needs to be able to SSH into each machine in air-gapped environment. -* [Optional] A media such as USB drive to copy the offline OEK package to the air-gapped environment if there is no connection from online node. -* All the nodes in air-gapped environment must be able to SSH to each other without requiring password input, see [getting-started.md](https://github.com/open-ness/ido-specs/blob/master/doc/getting-started/network-edge/controller-edge-node-setup.md#exchanging-ssh-keys-between-hosts). -* The control plane node needs to be able to SSH itself. -* The time and date of the nodes in offline environment is manually synchronized by the cluster's admin. -* User provided files - OPAE_SDK_1.3.7-5_el7.zip and syscfg_package.zip - -## Creating the offline package from an online node - -To create the offline package the user must have an access to an online node from which the offline package creator can download all necessary files and build Docker images. The list of files to be downloaded/build is provided in a form of a package definition list (Only package definition list for "flexran" flavor of OpenNESS is provided at the time of writing). Various categories of files to be downloaded are provided within this list including: RPMs, PIP pacakges, Helm charts, Dockerfiles, Go modules, and miscellaneous downloads. According to the category of a file the logic of offline package creator script will handle the download/build accordingly. Some files such as proprietary packages need to be provided by user in specified directories (see following steps). Once the offline package creator collects all necessary components it will pack them into an archive and then place them in appropriate place within the OpenNESS Experience Kits directory. Once the packages are archived the OpenNESS Experience Kits are ready to be deployed in air-gapped environment. The following diagram illustrates the workflow of the offline package creator. Additional information regarding the offline package creator can be found in the [README.md file](https://github.com/open-ness/openness-experience-kits/blob/master/offline_package_creator/README.md). - -Figure 3. Offline package creator workflow - - -To run the offline package creator run the following steps (user should not be a "root" user but does need "sudo" privileges to create the package, RT components will require installation of RT kernel on the node by the OPC): - -Clone the OpenNESS Experience Kits repo to an online node: - -```shell -# https://github.com/open-ness/ido-openness-experience-kits.git -``` - -Navigate to offline package creator directory: - -```shell -# cd ido-openness-experience-kits/oek/offline_package_creator/ -``` - -Create a directory from which user provided files can be accessed: - -```shell -# mkdir /->**NOTE**: The automatic inclusion of the `common` role should be handled by Ansible mechanisms (e.g., usage of meta's `dependencies` or `include_role` module) -* Name of the main role must be added to the `available_kubernetes_cnis` variable in `roles/kubernetes/cni/defaults/main.yml`. -* If additional requirements must checked before running the playbook (to not have errors during execution), they can be placed in the `roles/kubernetes/cni/tasks/precheck.yml` file, which is included as a pre_task in plays for both Edge Controller and Edge Node.
-The following are basic prechecks that are currently executed: - * Check if any CNI is requested (i.e., `kubernetes_cni` is not empty). - * Check if `sriov` is not requested as primary (first on the list) or standalone (only on the list). - * Check if `kubeovn` is requested as a primary (first on the list). - * Check if the requested CNI is available (check if some CNI is requested that isn't present in the `available_kubernetes_cnis` list). -* CNI roles should be as self-contained as possible (unless necessary, CNI-specific tasks should not be present in `kubernetes/{controlplane,node,common}` or `openness/network_edge/{controlplane,node}`). -* If the CNI needs a custom OpenNESS service (e.g., Interface Service in case of `kube-ovn`), it can be added to the `openness/network_edge/{controlplane,node}`.
- Preferably, such tasks would be contained in a separate task file (e.g., `roles/openness/controlplane/tasks/kube-ovn.yml`) and executed only if the CNI is requested. For example: - ```yaml - - name: deploy interface service for kube-ovn - include_tasks: kube-ovn.yml - when: "'kubeovn' in kubernetes_cnis" - ``` -* If the CNI is used as an additional CNI (with Multus\*), the network attachment definition must be supplied ([refer to Multus docs for more info](https://github.com/intel/multus-cni/blob/master/doc/quickstart.md#storing-a-configuration-as-a-custom-resource)). diff --git a/doc/orchestration/openness-helm.md b/doc/orchestration/openness-helm.md index 393cf68d..91a59c8d 100644 --- a/doc/orchestration/openness-helm.md +++ b/doc/orchestration/openness-helm.md @@ -12,7 +12,7 @@ Copyright (c) 2020 Intel Corporation - [References](#references) ## Introduction -Helm is a package manager for Kubernetes\*. It allows developers and operators to easily package, configure, and deploy applications and services onto Kubernetes clusters. For details refer to the [Helm Website](https://helm.sh). With OpenNESS, Helm is used to extend the [OpenNESS Experience Kits](https://github.com/open-ness/openness-experience-kits) Ansible\* playbooks to deploy Kubernetes packages. Helm adds considerable flexibility. It enables users to upgrade an existing installation without requiring a re-install. It provides the option to selectively deploy individual microservices if a full installation of OpenNESS is not needed. And it provides a standard process to deploy different applications or network functions. This document aims to familiarize the user with Helm and provide instructions on how to use the specific Helm charts available for OpenNESS. +Helm is a package manager for Kubernetes\*. It allows developers and operators to easily package, configure, and deploy applications and services onto Kubernetes clusters. For details refer to the [Helm Website](https://helm.sh). With OpenNESS, Helm is used to extend the [Converged Edge Experience Kits](https://github.com/open-ness/converged-edge-experience-kits) Ansible\* playbooks to deploy Kubernetes packages. Helm adds considerable flexibility. It enables users to upgrade an existing installation without requiring a re-install. It provides the option to selectively deploy individual microservices if a full installation of OpenNESS is not needed. And it provides a standard process to deploy different applications or network functions. This document aims to familiarize the user with Helm and provide instructions on how to use the specific Helm charts available for OpenNESS. ## Architecture The below figure shows the architecture for the OpenNESS Helm in this document. @@ -22,7 +22,7 @@ _Figure - Helm Architecture in OpenNESS_ ## Helm Installation -Helm 3 is used for OpenNESS. The installation is automatically conducted by the [OpenNESS Experience Kits](https://github.com/open-ness/openness-experience-kits) Ansible playbooks as below: +Helm 3 is used for OpenNESS. The installation is automatically conducted by the [Converged Edge Experience Kits](https://github.com/open-ness/converged-edge-experience-kits) Ansible playbooks as below: ```yaml - role: kubernetes/helm ``` @@ -39,7 +39,7 @@ OpenNESS provides the following helm charts: - CNI plugins including Multus\* and SRIOV CNI - Video analytics service - 5G control plane pods including AF, NEF, OAM, and CNTF -> **NOTE**: NFD, CMK, Prometheus, NodeExporter, and Grafana leverage existing third-party helm charts: [Container Experience Kits](https://github.com/intel/container-experience-kits) and [Helm GitHub\* Repo](https://github.com/helm/charts). For other helm charts, [OpenNESS Experience Kits](https://github.com/open-ness/openness-experience-kits) Ansible playbooks perform automatic charts generation and deployment. +> **NOTE**: NFD, CMK, Prometheus, NodeExporter, and Grafana leverage existing third-party helm charts: [Container Experience Kits](https://github.com/intel/container-experience-kits) and [Helm GitHub\* Repo](https://github.com/helm/charts). For other helm charts, [Converged Edge Experience Kits](https://github.com/open-ness/converged-edge-experience-kits) Ansible playbooks perform automatic charts generation and deployment. - Sample applications, network functions, and services that can be deployed and verified on the OpenNESS platform: - Applications @@ -49,11 +49,11 @@ OpenNESS provides the following helm charts: - [Telemetry Sample Application Helm Charts](https://github.com/open-ness/edgeapps/tree/master/applications/telemetry-sample-app) - [EIS Sample Application Helm Charts](https://github.com/open-ness/edgeapps/tree/master/applications/eis-experience-kit) - Network Functions - - [FlexRAN Helm Charts](https://github.com/open-ness/edgeapps/tree/master/network-functions/ran/charts/flexran) + - [FlexRAN Helm Charts](https://github.com/open-ness/edgeapps/tree/master/network-functions/ran/charts/du-dev) - [xRAN Helm Charts](https://github.com/open-ness/edgeapps/tree/master/network-functions/xran/helmcharts/xranchart) - [UPF Helm Charts](https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/charts/upf) -The EPA, Telemetry, and k8s plugins helm chart files will be saved in a specific directory on the OpenNESS controller. To modify the directory, change the following variable `ne_helm_charts_default_dir` in the `group_vars/all/10-default.yml` file: +The EPA, Telemetry, and k8s plugins helm chart files will be saved in a specific directory on the OpenNESS controller. To modify the directory, change the following variable `ne_helm_charts_default_dir` in the `inventory/default/group_vars/all/10-open.yml` file: ```yaml ne_helm_charts_default_dir: /opt/openness/helm-charts/ ``` diff --git a/doc/reference-architectures/CERA-5G-On-Prem.md b/doc/reference-architectures/CERA-5G-On-Prem.md index 399b52a8..6ebbbb43 100644 --- a/doc/reference-architectures/CERA-5G-On-Prem.md +++ b/doc/reference-architectures/CERA-5G-On-Prem.md @@ -1,6 +1,6 @@ ```text SPDX-License-Identifier: Apache-2.0 -Copyright (c) 2020 Intel Corporation +Copyright (c) 2020-2021 Intel Corporation ``` # Converged Edge Reference Architecture 5G On Premises Edge @@ -11,7 +11,7 @@ The Converged Edge Reference Architectures (CERA) are a set of pre-integrated HW - [CERA 5G On Prem OpenNESS Configuration](#cera-5g-on-prem-openness-configuration) - [CERA 5G On Prem Deployment Architecture](#cera-5g-on-prem-deployment-architecture) - [CERA 5G On Prem Experience Kit Deployments](#cera-5g-on-prem-experience-kit-deployments) - - [Edge Service Applications Supported on CERA 5G On Prem](#edge-service-applications-supported-on-cera-5g-on-prem) + - [Edge Service Applications Supported by CERA 5G On Prem](#edge-service-applications-supported-by-cera-5g-on-prem) - [OpenVINO™](#openvino) - [Edge Insights Software](#edge-insights-software) - [CERA 5G On Prem Hardware Platform](#cera-5g-on-prem-hardware-platform) @@ -21,7 +21,6 @@ The Converged Edge Reference Architectures (CERA) are a set of pre-integrated HW - [BIOS Setup](#bios-setup) - [Setting up Machine with Ansible](#setting-up-machine-with-ansible) - [Steps to be performed on the machine, where the Ansible playbook is going to be run](#steps-to-be-performed-on-the-machine-where-the-ansible-playbook-is-going-to-be-run) - - [CERA 5G On Premise Experience Kit Deployment](#cera-5g-on-premise-experience-kit-deployment) - [5G Core Components](#5g-core-components) - [dUPF](#dupf) - [Overview](#overview) @@ -231,26 +230,59 @@ The BIOS settings on the edge node must be properly set in order for the OpenNES git submodule update --init --recursive ``` -4. Provide target machines IP addresses for OpenNESS deployment in `ido-converged-edge-experience-kits/openness_inventory.ini`. For Singlenode setup, set the same IP address for both `controller` and `node01`, the line with `node02` should be commented by adding # at the beginning. -Example: - ```ini - [all] - controller ansible_ssh_user=root ansible_host=192.168.1.43 # First server NE - node01 ansible_ssh_user=root ansible_host=192.168.1.43 # First server NE - ; node02 ansible_ssh_user=root ansible_host=192.168.1.12 - ``` - At that stage provide IP address only for `CERA 5G NE` server. - - If the GMC device is available, the node server can be synchronized. In the `ido-converged-edge-experience-kits/openness_inventory.ini`, `node01` should be added to `ptp_slave_group`. The default value `controller` for `[ptp_master]` should be removed or commented. - ```ini - [ptp_master] - #controller - - [ptp_slave_group] - node01 +4. Provide target machines IP addresses for CEEK deployment in `ido-converged-edge-experience-kits/inventory.yml`. For Singlenode setup, set the same IP address for both `controller` and `node01`. In the same file define the details for Central Office cluster deployment. + Example: + ```yaml + all: + vars: + cluster_name: on_premises_cluster + flavor: cera_5g_on_premise + single_node_deployment: true + limit: + controller_group: + hosts: + controller: + ansible_host: 172.16.0.1 + ansible_user: root + edgenode_group: + hosts: + node01: + ansible_host: 172.16.0.1 + ansible_user: root + edgenode_vca_group: + hosts: + ptp_master: + hosts: + ptp_slave_group: + hosts: + + --- + all: + vars: + cluster_name: central_office_cluster + flavor: cera_5g_central_office + single_node_deployment: true + limit: + controller_group: + hosts: + co_controller: + ansible_host: 172.16.1.1 + ansible_user: root + edgenode_group: + hosts: + co_node1: + ansible_host: 172.16.1.1 + ansible_user: root + edgenode_vca_group: + hosts: + ptp_master: + hosts: + ptp_slave_group: + hosts: + ``` -5. Edit `ido-converged-edge-experience-kits/openness/group_vars/all/10-open.yml` and provide some correct settings for deployment. +5. Edit `ido-converged-edge-experience-kits/inventory/default/group_vars/all/10-open.yml` and provide some correct settings for deployment. Git token. ```yaml @@ -280,213 +312,115 @@ Example: ntp_servers: ['ntp.server.com'] ``` -6. Edit file `ido-converged-edge-experience-kits/openness/flavors/cera_5g_on_premise/edgenode_group.yml` and provide correct CPU settings. +6. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml` and provide On Premise deployment configuration. + Choose Edge Application that will be deployed: ```yaml - tuned_vars: | - isolated_cores=2-23,26-47 - nohz=on - nohz_full=2-23,26-47 - - # CPUs to be isolated (for RT procesess) - cpu_isol: "2-23,26-47" - # CPUs not to be isolate (for non-RT processes) - minimum of two OS cores necessary for controller - cpu_os: "0-1,24-25" + # Choose which demo will be launched: `eis` or `openvino` + # To do not deploy any demo app, refer to `edgeapps_deployment_enable` variable + deploy_demo_app: "openvino" ``` - If a GMC is connected to the setup, then node server synchronization can be enabled inside ido-converged-edge-experience-kits/openness/flavors/cera_5g_on_premise/edgenode_group.yml file. + If OpenVINO was chosen as Edge Application, set the options: ```yaml - ptp_sync_enable: true + model: "pedestrian-detection-adas-0002" # model name used by demo application + display_host_ip: "" # update ip for visualizer HOST GUI. + save_video: "enable" # enable saving the output video to file + target_device: "CPU" # device which will be used for video processing, currently only CPU is supported ``` -7. Edit file `ido-converged-edge-experience-kits/openness/flavors/cera_5g_on_premise/controller_group.yml` and provide names of `network interfaces` that are connected to second server and number of VF's to be created. - + Set interface name for fronthaul connection: ```yaml - sriov: - network_interfaces: {eno1: 5, eno2: 10} + ## Interface logical name (PF) used for fronthaul + fronthaul_if_name: "enp184s0f0" ``` -8. Edit file `ido-converged-edge-experience-kits/openness/x-oek/oek/host_vars/node01.yml` if a GMC is connected and the node server should be synchronized. - - For single node setup (this is the default mode for CERA), `ptp_port` keeps the host's interface connected to Grand Master, e.g.: + Set interface name for connection to UPF and AMF-SMF: ```yaml - ptp_port: "eno3" + # PF interface name of N3, N4, N6, N9 created VFs + host_if_name_cn: "eno1" ``` - Variable `ptp_network_transport` keeps network transport for ptp. Choose `"-4"` for default CERA setup. The `gm_ip` variable should contain the GMC's IP address. The Ansible scripts set the IP on the interface connected to the GMC, according to the values in the variables `ptp_port_ip` and `ptp_port_cidr`. - ```yaml - # Valid options: - # -2 Select the IEEE 802.3 network transport. - # -4 Select the UDP IPv4 network transport. - ptp_network_transport: "-4" - - # Grand Master IP, e.g.: - # gm_ip: "169.254.99.9" - gm_ip: "169.254.99.9" - - # - ptp_port_ip contains a static IP for the server port connected to GMC, e.g.: - # ptp_port_ip: "169.254.99.175" - # - ptp_port_cidr - CIDR for IP from, e.g.: - # ptp_port_cidr: "24" - ptp_port_ip: "169.254.99.175" - ptp_port_cidr: "24" - ``` +7. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/controller_group.yml` -9. Execute the `deploy_openness_for_cera.sh` script in `ido-converged-edge-experience-kits` to start OpenNESS platform deployment process by running the following command: - ```shell - ./deploy_openness_for_cera.sh cera_5g_on_premise - ``` - Note: This might take few hours. - -10. After a successful OpenNESS deployment, edit again `ido-converged-edge-experience-kits/openness_inventory.ini`, change IP address to `CERA 5G CN` server. - ```ini - [all] - controller ansible_ssh_user=root ansible_host=192.168.1.109 # Second server CN - node01 ansible_ssh_user=root ansible_host=192.168.1.109 # Second server CN - ; node02 ansible_ssh_user=root ansible_host=192.168.1.12 - ``` - Then run `deploy_openness_for_cera.sh` again. - ```shell - ./deploy_openness_for_cera.sh - ``` - All settings in `ido-converged-edge-experience-kits/openness/group_vars/all/10-open.yml` are the same for both servers. - - For `CERA 5G CN` server disable synchronization with GMC inside `ido-converged-edge-experience-kits/openness/flavors/cera_5g_on_premise/edgenode_group.yml` file. + Provide names of `network interfaces` that are connected to the Central Office cluster and number of VF's to be created. ```yaml - ptp_sync_enable: false - ``` - -11. When both servers have deployed OpenNess, login to `CERA 5G CN` server and generate `RSA ssh key`. It's required for AMF/SMF VM deployment. - ```shell - ssh-keygen -t rsa - # Press enter key to apply default values + sriov: + network_interfaces: {eno1: 5, enp184s0f0: 10} ``` -12. The full setup is now ready for CERA deployment. - -### CERA 5G On Premise Experience Kit Deployment -The following prerequisites should be met for CERA deployment. -1. CentOS should use the following kernel and have no newer kernels installed: - * `3.10.0-1127.19.1.rt56.1116.el7.x86_64` on Near Edge server. - * `3.10.0-1127.el7.x86_64` on Core Network server. +8. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/edgenode_group.yml` -2. Edit file `ido-converged-edge-experience-kits/cera_config.yaml` and provide correct settings: - - Git token - ```yaml - git_repo_token: "your git token" - ``` - Decide which demo application should be launched - ```yaml - # choose which demo will be launched: `eis` or `openvino` - deploy_app: "eis" - ``` - EIS release package location + Set up CPUs isolation according to your hardware ```yaml - # provide EIS release package archive absolute path - eis_release_package_path: "" + # Variables applied with the profile + tuned_vars: | + isolated_cores=1-16,25-40 + nohz=on + nohz_full=1-16,25-40 + + # CPUs to be isolated (for RT procesess) + cpu_isol: "1-16,25-40" + # CPUs not to be isolate (for non-RT processes) - minimum of two OS cores necessary for controller + cpu_os: "0,17-23,24,41-47" ``` - [OpenVino](#OpenVINO) settings, if OpenVino app was set as active demo application + + Set up hugepages settings ```yaml - display_host_ip: "" # update ip for visualizer HOST GUI. - save_video: "enable" + # Size of a single hugepage (2M or 1G) + hugepage_size: "1G" + # Amount of hugepages + hugepage_amount: "40" ``` - Proxy settings - ```yaml - # Setup proxy on the machine - required if the Internet is accessible via proxy - proxy_os_enable: true - # Clear previous proxy settings - proxy_os_remove_old: true - # Proxy URLs to be used for HTTP, HTTPS and FTP - proxy_os_http: "http://proxy.example.org:3129" - proxy_os_https: "http://proxy.example.org:3128" - proxy_os_ftp: "http://proxy.example.org:3128" - proxy_os_noproxy: "127.0.0.1,localhost,192.168.1.0/24" - # Proxy to be used by YUM (/etc/yum.conf) - proxy_yum_url: "{{ proxy_os_http }}" - ``` - See [more details](#dUPF) for dUPF configuration + +9. Set all necessary settings for `CERA 5G CO` in `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/all.yml`. + ```yaml - # Define PCI addresses (xxxx:xx:xx.x format) for i-upf - n3_pci_bus_address: "0000:19:0a.0" - n4_n9_pci_bus_address: "0000:19:0a.1" - n6_pci_bus_address: "0000:19:0a.2" - - # Define VPP VF interface names for i-upf - n3_vf_interface_name: "VirtualFunctionEthernet19/a/0" - n4_n9_vf_interface_name: "VirtualFunctionEthernet19/a/1" - n6_vf_interface_name: "VirtualFunctionEthernet19/a/2" - - # PF interface name of N3 created VF - host_if_name_N3: "eno2" # PF interface name of N4, N6, N9 created VFs - host_if_name_N4_N6_n9: "eno2" + host_if_name_cn: "eno1" ``` - [gNodeB](#gNodeB) configuration - ```yaml - ## gNodeB related config - gnodeb_fronthaul_vf1: "0000:65:02.0" - gnodeb_fronthaul_vf2: "0000:65:02.1" - gnodeb_fronthaul_vf1_mac: "ac:1f:6b:c2:48:ad" - gnodeb_fronthaul_vf2_mac: "ac:1f:6b:c2:48:ab" +10. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/controller_group.yml` - n2_gnodeb_pci_bus_address: "0000:19:0a.3" - n3_gnodeb_pci_bus_address: "0000:19:0a.4" - - fec_vf_pci_addr: "0000:b8:00.1" - - # DPDK driver used (vfio-pci/igb_uio) to VFs bindings - dpdk_driver_gnodeb: "igb_uio" - - ## ConfigMap vars - - fronthaul_if_name: "enp101s0f0" - ``` - Settings for `CERA 5G CN` + Provide names of `network interfaces` that are connected to the Near Edge cluster and number of VF's to be created. ```yaml - ## PSA-UPF vars - - # Define N4/N9 and N6 interface device PCI bus address - PCI_bus_address_N4_N9: '0000:19:0a.0' - PCI_bus_address_N6: '0000:19:0a.1' + sriov: + network_interfaces: {eno1: 5} + ``` - # 5gc binaries directory name - package_5gc_path: "/opt/amf-smf/" +11. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/edgenode_group.yml` - # vpp interface name as per setup connection - vpp_interface_N4_N9_name: 'VirtualFunctionEthernet19/a/0' - vpp_interface_N6_name: 'VirtualFunctionEthernet19/a/1' - ``` -3. If needed change additional settings for `CERA 5G NE` in `ido-converged-edge-experience-kits/host_vars/cera_5g_ne.yml`. + Set up CPUs isolation according to your hardware capabilities ```yaml - # DPDK driver used (vfio-pci/igb_uio) to VFs bindings - dpdk_driver_upf: "igb_uio" + # Variables applied with the profile + tuned_vars: | + isolated_cores=1-16,25-40 + nohz=on + nohz_full=1-16,25-40 - # Define path where i-upf is located on remote host - upf_binaries_path: "/opt/flexcore-5g-rel/i-upf/" + # CPUs to be isolated (for RT procesess) + cpu_isol: "1-16,25-40" + # CPUs not to be isolate (for non-RT processes) - minimum of two OS cores necessary for controller + cpu_os: "0,17-23,24,41-47" ``` - OpenVino model + + Set up hugepages settings ```yaml - model: "pedestrian-detection-adas-0002" - ``` -4. Build the following docker images required and provide necessary binaries. - - [dUPF](#dUPF) - - [UPF](#UPF) - - [AMF-SMF](#AMF-SMF) - - [gNB](#gNodeB) -5. Provide correct IP for target servers in file `ido-converged-edge-experience-kits/cera_inventory.ini` - ```ini - [all] - cera_5g_ne ansible_ssh_user=root ansible_host=192.168.1.109 - cera_5g_cn ansible_ssh_user=root ansible_host=192.168.1.43 - ``` -6. Deploy CERA Experience Kit + # Size of a single hugepage (2M or 1G) + hugepage_size: "1G" + # Amount of hugepages + hugepage_amount: "8" + +12. Deploy Converged Edge Experience Kit (Near Edge and Central Office clusters simultaneously) + + Silent deployment: ```shell - ./deploy_cera.sh + python ./deploy.py ``` + > NOTE: In multicluster deployment logs are hidden by default. To check the logs `tail` tool can be used on deployment log files. + ## 5G Core Components This section describes in details how to build particular images and configure ansible for deployment. @@ -502,18 +436,20 @@ The `CERA dUPF` component is deployed on `CERA 5G Near Edge (cera_5g_ne)` node. ##### Prerequisites -To deploy dUPF correctly, one needs to provide Docker image to Docker repository on the target node. There is a script on the `open-ness/eddgeapps/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. +To deploy dUPF correctly, one needs to provide Docker image to Docker repository on the target node. There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. + +```sh +./build_image.sh -b i-upf -i i-upf +``` ##### Settings -The following variables need to be defined in `cera_config.yaml` +The following variables need to be defined in `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml` ```yaml -n3_pci_bus_address: "" - PCI bus address of VF, which is used for N3 interface by dUPF -n4_n9_pci_bus_address: "" - PCI bus address of VF, which is used for N4 and N9 interface by dUPF -n6_pci_bus_address: "" - PCI bus address of VF, which is used for N6 interface by dUPF +## Interface logical name (PF) used for fronthaul +fronthaul_if_name: "enp184s0f0" -n3_vf_interface_name: "" - name of VF, which is used for N3 interface by dUPF -n4_n9_vf_interface_name: "" - name of VF, which is used for N4 and N9 interface by dUPF -n6_vf_interface_name: "" - name of VF, which is used for N6 interface by dUPF +# PF interface name of N3, N4, N6, N9 created VFs +host_if_name_cn: "eno1" ``` ##### Configuration @@ -524,22 +460,23 @@ The dUPF is configured automatically during the deployment. The `User Plane Function (UPF)` is a part of 5G Core Network, it is responsible for packets routing. It has 2 separate interfaces for `N4/N9` and `N6` data lines. `N4/N9` interface is used for connection with `dUPF` and `AMF/SMF` (locally). `N6` interface is used for connection with `EDGE-APP`, `dUPF` and `Remote-DN` (locally). -The CERA UPF component is deployed on `CERA 5G Core Network (cera_5g_cn)` node. It is deployed as a POD - during deployment of CERA 5G On Prem automatically. +The CERA UPF component is deployed on `CERA 5G Central Office` node. It is deployed as a POD - during deployment of CERA 5G Central Office flavor automatically. #### Deployment ##### Prerequisites -To deploy `UPF` correctly one needs to provide a Docker image to Docker Repository on target nodes. There is a script on the `open-ness/eddgeapps/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. +To deploy `UPF` correctly one needs to provide a Docker image to Docker Repository on target nodes. There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. + +```sh +./build_image.sh -b psa-upf -i psa-upf +``` ##### Settings -The following variables need to be defined in the `cera_config.yaml` -```yaml -PCI_bus_address_N4_N9: "" - PCI bus address of VF, which is used for N4 and N9 interface by UPF -PCI_bus_address_N6: "" - PCI bus address of VF, which is used for N6 interface by UPF +Update interface name in file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/all.yml` that is used for connection to Near Edge cluster (dUPF). -vpp_interface_N4_N9_name: "" - name of VF, which is used for N4 and N9 interface by UPF -vpp_interface_N6_name: "" - name of VF, which is used for N6 interface by UPF +```yaml +host_if_name_cn: "eno1" ``` ##### Configuration @@ -556,21 +493,21 @@ The CERA `AMF-SMF` component is deployed on `CERA 5G Core Network (cera_5g_cn)` #### Deployment ##### Prerequisites -To deploy `AMF-SMF` correctly, one needs to provide a Docker image to Docker Repository on target machine(cera_5g_cn). There is a script on the `open-ness/eddgeapps/network-functions/core-network/5G/AMF-SMF` repository provided by CERA, which builds the image automatically. - -##### Settings +To deploy `AMF-SMF` correctly, one needs to provide a Docker image to Docker Repository on target machine(cera_5g_co). There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/AMF_SMF` repository provided by CERA, which builds the image automatically. -Following variables need to be defined in `cera_config.yaml` -```yaml -# 5gc binaries directory name -package_5gc_path: "/opt/amf-smf/" +```sh +./build_image.sh -b amf-smf ``` +##### Settings +No special settings are required for AMF-SMF deployment. + ##### Configuration The `AMF-SMF` is configured automatically during the deployment. ### Remote-DN + #### Overview Remote Data Network is component, which represents `“internet”` in networking. CERA Core Network manages which data should apply to `Near Edge Application(EIS/OpenVINO)` or go further to the network. @@ -590,12 +527,12 @@ Deployment of Local-DN is completely automated, so there is no need to set or co ### OpenVINO #### Settings -In the `cera_config.yaml` file can be chosen for which application should be built and deployed. Set a proper value for the deploy_app variable. +In the `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml` file can be chosen which application should be built and deploy. Set a proper value for the deploy_app variable. ```yaml -deploy_app: "" - Type openvino if OpenVINO demo should be launched. +deploy_demo_app: "" - Type openvino if OpenVINO demo should be launched. ``` -Several variables must be set in the file `host_vars/cera_5g_ne.yml`: +Several variables must be set in the file `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml`: ```yaml model: "pedestrian-detection-adas-0002" - Model for which the OpenVINO demo will be run. Models which can be selected: pedestrian-detection-adas-0002, pedestrian-detection-adas-binary-0001, pedestrian-and-vehicle-detector-adas-0001, vehicle-detection-adas-0002, vehicle-detection-adas-binary-0001, person-vehicle-bike-detection-crossroad-0078, person-vehicle-bike-detection-crossroad-1016, person-reidentification-retail-0031, person-reidentification-retail-0248, person-reidentification-retail-0249, person-reidentification-retail-0300, road-segmentation-adas-0001 @@ -603,7 +540,7 @@ save_video: "enable" - For value "enable" the output will be written to /root/sa ``` #### Deployment -After running the `deploy_cera.sh` script, pod ov-openvino should be available on `cera_5g_ne` machine. The status of the ov-openvino pod can be checked by use: +After running the `deploy.py` script, pod ov-openvino should be available on Near Edge cluster. The status of the ov-openvino pod can be checked by use: ```shell kubectl -n openvino get pods -o wide ``` @@ -612,7 +549,7 @@ Immediately after creating, the ov-openvino pod will wait for input streaming. I #### Streaming Video to OpenVINO™ pod should be streamed to IP `192.168.1.101` and port `5000`. Make sure that the pod with OpenVINO™ is visible from your streaming machine. In the simplest case, the video can be streamed from the same machine where pod with OpenVINO™ is available. -Output will be saved to the `saved_video/ov-output.mjpeg` file (`save_video` variable in the `host_vars/cera_5g_ne.yml` should be set to `"enable"` and should be not changed). +Output will be saved to the `saved_video/ov-output.mjpeg` file (`save_video` variable in the `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml` should be set to `"enable"` and should be not changed). Streaming is possible from a file or from a camera. For continuous and uninterrupted streaming of a video file, the video file can be streamed in a loop. An example of a Bash file for streaming is shown below. ```shell @@ -647,35 +584,19 @@ For more details about `eis-experience-kit` check [README.md](https://github.com ### gNodeB #### Overview -`gNodeB` is a part of 5G Core Architecture and is deployed on `CERA 5G Nere Edge (cera_5g_ne)` node. +`gNodeB` is a part of 5G Core Architecture and is deployed on `CERA 5G On Premise (cera_5g_ne)` node. #### Deployment #### Prerequisites -To deploy `gNodeB` correctly it is required to provide a Docker image to Docker Repository on target machine(cera_5g_ne). There is a script on the `open-ness/eddgeapps/network-functions/ran/5G/gnb` repository provided by CERA, which builds the image automatically. For `gNodeB` deployment FPGA card is required PAC N3000 and also QAT card. +To deploy `gNodeB` correctly it is required to provide a Docker image to Docker Repository on target machine(cera_5g_ne). There is a script on the `open-ness/eddgeapps/network-functions/ran/5G/gnb` repository provided by CERA, which builds the image automatically. For `gNodeB` deployment FPGA card is required PAC N3000 and also QAT card. #### Settings -The following variables need to be defined in `cera_config.yaml` +The following variables need to be defined in `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/all.yml` ```yaml -## gNodeB related config -# Fronthaul require two VFs -gnodeb_fronthaul_vf1: "0000:65:02.0" - PCI bus address of VF, which is used as fronthaul -gnodeb_fronthaul_vf2: "0000:65:02.1" - PCI bus address of VF, which is used as fronthaul - -gnodeb_fronthaul_vf1_mac: "ac:1f:6b:c2:48:ad" - MAC address which will be set on the first VF during deployment -gnodeb_fronthaul_vf2_mac: "ac:1f:6b:c2:48:ab" - MAC address which will be set on the second VF during deployment - -n2_gnodeb_pci_bus_address: "0000:19:0a.3" - PCI bus address of VF, which is used for N2 interface -n3_gnodeb_pci_bus_address: "0000:19:0a.4" - PCI bus address of VF, which is used for N3 interface - -fec_vf_pci_addr: "0000:b8:00.1" - PCI bus address of VF, which is assigned to FEC PAC N3000 accelerator - -# DPDK driver used (vfio-pci/igb_uio) to VFs bindings -dpdk_driver_gnodeb: "igb_uio" - driver for binding interfaces - -## ConfigMap vars -fronthaul_if_name: "enp101s0f0" - name of fronthaul interface +## Interface logical name (PF) used for fronthaul +fronthaul_if_name: "enp101s0f0" ``` #### Configuration @@ -726,19 +647,20 @@ GMC must be properly configured and connected to the server's ETH port. #### Settings If the GMC has been properly configured and connected to the server then the node server can be synchronized. -In the `ido-converged-edge-experience-kits/openness_inventory.ini` file, `node01` should be added to `ptp_slave_group` and the content inside the `ptp_master` should be empty or commented. -```ini -[ptp_master] -#controller +In the `ido-converged-edge-experience-kits/inventory.yml` file, `node01` should be added to `ptp_slave_group` and the content inside the `ptp_master` should be empty or commented. +```yaml +ptp_master: + hosts: -[ptp_slave_group] -node01 +ptp_slave_group: + hosts: + node01 ``` -Server synchronization can be enabled inside `ido-converged-edge-experience-kits/openness/flavors/cera_5g_on_premise/edgenode_group.yml` file. +Server synchronization can be enabled inside `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/edgenode_group.yml` file. ```yaml ptp_sync_enable: true ``` -Edit file `ido-converged-edge-experience-kits/openness/x-oek/oek/host_vars/node01.yml` if a GMC is connected and the node server should be synchronized. +Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_on_premise/edgenode_group.yml` if a GMC is connected and the node server should be synchronized. For single node setup (this is the default mode for CERA), `ptp_port` keeps the host's interface connected to Grand Master, e.g.: ```yaml @@ -806,6 +728,7 @@ CERA 5G On Premises deployment provides a reference implementation of how to use | CERA | Converged Edge Reference Architecture | | CN | Core Network | | CNF | Container Network Function | +| CO | Central Office | | CommSPs | Communications Service Providers | | DPDK | Data Plane Developer Kit | | eNB | e-NodeB | diff --git a/doc/reference-architectures/CERA-Near-Edge.md b/doc/reference-architectures/CERA-Near-Edge.md index b422da83..f96097b8 100644 --- a/doc/reference-architectures/CERA-Near-Edge.md +++ b/doc/reference-architectures/CERA-Near-Edge.md @@ -23,10 +23,8 @@ Reference architecture combines wireless and high performance compute for IoT, A - [Setting up target platform before deployment](#setting-up-target-platform-before-deployment) - [BIOS Setup](#bios-setup) - [Manual setup](#manual-setup) - - [Setup through the CERA deployment](#setup-through-the-cera-deployment) - [Setting up machine with Ansible](#setting-up-machine-with-ansible) - [Steps to be performed on the machine, where the Ansible playbook is going to be run](#steps-to-be-performed-on-the-machine-where-the-ansible-playbook-is-going-to-be-run) - - [CERA Near Edge Experience Kit Deployment](#cera-near-edge-experience-kit-deployment) - [5G Core Components](#5g-core-components) - [dUPF](#dupf) - [Overview](#overview) @@ -46,7 +44,6 @@ Reference architecture combines wireless and high performance compute for IoT, A - [Prerequisites](#prerequisites-2) - [Settings](#settings-2) - [Configuration](#configuration-2) - - [How to prepare image](#how-to-prepare-image) - [Remote-DN](#remote-dn) - [Overview](#overview-3) - [Prerequisites](#prerequisites-3) @@ -241,43 +238,11 @@ There are two possibilities to change BIOS settings. The most important paramete #### Manual setup Reboot platform, go to the BIOS setup during server boot process and set correct options. -#### Setup through the CERA deployment -Bios will be set automatically during CERA deployment according to the provided settings. -* Provide correct `bios_settings.ini` file for `Intel SYSCFG utility` and store it in `ido-converged-edge-experience-kits/roles/bios_setup/files/` -* Set correct name of variable `biosconfig_local_path` in file: `ido-converged-edge-experience-kits/cera_5g_near_edge_deployment.yml` for both hosts. - ```yaml - # NE Server - - role: bios_setup - vars: - biosconfig_local_path: "bios_config_cera_5g_ne.ini" - when: update_bios_ne | default(False) - ``` - ```yaml - # CN Server - - role: bios_setup - vars: - biosconfig_local_path: "bios_config_cera_5g_cn.ini" - when: update_bios_cn | default(False) - ``` -* Change variable to `True` in `ido-converged-edge-experience-kits/host_vars/cera_5g_cn.yml` and in `ido-converged-edge-experience-kits/host_vars/cera_5g_ne.yml` - - ```yaml - # Set True for bios update - update_bios_cn: True - ``` - ```yaml - # Set True for bios update - update_bios_ne: True - ``` -> NOTE: It's important to have correct bios.ini file with settings generated on the particular server. There are some unique serial numbers assigned to the server. - -More information: [BIOS and Firmware Configuration on OpenNESS Platform](https://www.openness.org/docs/doc/enhanced-platform-awareness/openness-bios) - ### Setting up machine with Ansible #### Steps to be performed on the machine, where the Ansible playbook is going to be run -1. Copy SSH key from machine, where the Ansible playbook is going to be run, to the target machine. Example commands: +1. Copy SSH key from machine, where the Ansible playbook is going to be run, to the target machines. Example commands: > NOTE: Generate ssh key if is not present on the machine: `ssh-keygen -t rsa` (Press enter key to apply default values) Do it for each target machine @@ -299,17 +264,60 @@ More information: [BIOS and Firmware Configuration on OpenNESS Platform](https:/ git submodule update --init --recursive ``` -4. Provide target machines IP addresses for OpenNESS deployment in `ido-converged-edge-experience-kits/openness_inventory.ini`. For Singlenode setup, set the same IP address for both `controller` and `node01`, the line with `node02` should be commented by adding # at the beginning. -Example: - ```ini - [all] - controller ansible_ssh_user=root ansible_host=192.168.1.43 # First server NE - node01 ansible_ssh_user=root ansible_host=192.168.1.43 # First server NE - ; node02 ansible_ssh_user=root ansible_host=192.168.1.12 - ``` - At that stage provide IP address only for `CERA 5G NE` server. +4. Provide target machines IP addresses for CEEK deployment in `ido-converged-edge-experience-kits/inventory.yml`. For Singlenode setup, set the same IP address for both `controller` and `node01`. In the same file define the details for Central Office cluster deployment. -5. Edit `ido-converged-edge-experience-kits/openness/group_vars/all/10-open.yml` and provide some correct settings for deployment. + Example: + ```yaml + all: + vars: + cluster_name: near_edge_cluster # NOTE: Use `_` instead of spaces. + flavor: cera_5g_near_edge # NOTE: Flavors can be found in `flavors` directory. + single_node_deployment: true # Request single node deployment (true/false). + limit: # Limit ansible deployment to certain inventory group or hosts + controller_group: + hosts: + controller: + ansible_host: 172.16.0.1 + ansible_user: root + edgenode_group: + hosts: + node01: + ansible_host: 172.16.0.1 + ansible_user: root + edgenode_vca_group: + hosts: + ptp_master: + hosts: + ptp_slave_group: + hosts: + + --- + all: + vars: + cluster_name: central_office_cluster # NOTE: Use `_` instead of spaces. + flavor: cera_5g_central_office # NOTE: Flavors can be found in `flavors` directory. + single_node_deployment: true # Request single node deployment (true/false). + limit: # Limit ansible deployment to certain inventory group or hosts + controller_group: + hosts: + co_controller: + ansible_host: 172.16.1.1 + ansible_user: root + edgenode_group: + hosts: + co_node1: + ansible_host: 172.16.1.1 + ansible_user: root + edgenode_vca_group: + hosts: + ptp_master: + hosts: + ptp_slave_group: + hosts: + + ``` + +5. Edit `ido-converged-edge-experience-kits/inventory/default/group_vars/all/10-open.yml` and provide some correct settings for deployment. Git token. ```yaml @@ -339,182 +347,86 @@ Example: ntp_servers: ['ger.corp.intel.com'] ``` -6. Edit file `ido-converged-edge-experience-kits/openness/flavors/cera_5g_near_edge/edgenode_group.yml` and provide correct CPU settings. +6. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_near_edge/all.yml` and provide Near Edge deployment configuration. + Choose Edge Application that will be deployed: ```yaml - tuned_vars: | - isolated_cores=1-16,25-40 - nohz=on - nohz_full=1-16,25-40 - # CPUs to be isolated (for RT procesess) - cpu_isol: "1-16,25-40" - # CPUs not to be isolate (for non-RT processes) - minimum of two OS cores necessary for controller - cpu_os: "0,17-23,24,41-47" + # Choose which demo will be launched: `eis` or `openvino` + # To do not deploy any demo app, refer to `edgeapps_deployment_enable` variable + deploy_demo_app: "openvino" ``` -7. Edit file `ido-converged-edge-experience-kits/openness/flavors/cera_5g_near_edge/controller_group.yml` and provide names of `network interfaces` that are connected to second server and number of VF's to be created. - + If OpenVINO was chosen as Edge Application, set the options: ```yaml - sriov: - network_interfaces: {eno1: 5, eno2: 2} - ``` - > NOTE: On various platform interfaces can have different name. For e.g `eth1` instead of `eno1`. Please verify interface name before deployment and do right changes. - -8. Execute the `deploy_openness_for_cera.sh` script in `ido-converged-edge-experience-kits` to start OpenNESS platform deployment process by running following command: - ```shell - ./deploy_openness_for_cera.sh cera_5g_near_edge - ``` - It might take few hours. - -9. After successful OpenNESS deployment, edit again `ido-converged-edge-experience-kits/openness_inventory.ini`, change IP address to `CERA 5G CN` server. - ```ini - [all] - controller ansible_ssh_user=root ansible_host=192.168.1.109 # Second server CN - node01 ansible_ssh_user=root ansible_host=192.168.1.109 # Second server CN - ; node02 ansible_ssh_user=root ansible_host=192.168.1.12 - ``` - Then run `deploy_openness_for_cera.sh` again. - ```shell - ./deploy_openness_for_cera.sh - ``` - All settings in `ido-converged-edge-experience-kits/openness/group_vars/all/10-open.yml` are the same for both servers. - -10. When both servers have deployed OpenNess, login to `CERA 5G CN` server and generate `RSA ssh key`. It's required for AMF/SMF VM deployment. - ```shell - ssh-keygen -t rsa - # Press enter key to apply default values + model: "pedestrian-detection-adas-0002" + display_host_ip: "" # update ip for visualizer HOST GUI. + save_video: "enable" + target_device: "CPU" ``` -11. Now full setup is ready for CERA deployment. - -### CERA Near Edge Experience Kit Deployment -For CERA deployment some prerequisites have to be fulfilled. - -1. CentOS should use kernel `kernel-3.10.0-957.el7.x86_64` and have no newer kernels installed. -2. Edit file `ido-converged-edge-experience-kits/group_vars/all.yml` and provide correct settings: - - Git token + Set interface name used for connection to Central Office cluster: ```yaml - git_repo_token: "your git token" - ``` - Decide which demo application should be launched - ```yaml - # choose which demo will be launched: `eis` or `openvino` - deploy_app: "eis" - ``` - EIS release package location - ```yaml - # provide EIS release package archive absolute path - eis_release_package_path: "" - ``` - AMF/SMF VM image location - ```yaml - # VM image path - vm_image_path: "/opt/flexcore-5g-rel/ubuntu_18.04.qcow2" + # PF interface name of N3, N4, N6, N9 created VFs + host_if_name_cn: "eno1" ``` -3. Edit file `ido-converged-edge-experience-kits/host_vars/localhost.yml` and provide correct proxy if is required. +7. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_near_edge/controller_group.yml` + Provide names of `network interfaces` that are connected to the second server and number of VF's to be created. ```yaml - ### Proxy settings - # Setup proxy on the machine - required if the Internet is accessible via proxy - proxy_os_enable: true - # Clear previous proxy settings - proxy_os_remove_old: true - # Proxy URLs to be used for HTTP, HTTPS and FTP - proxy_os_http: "http://proxy.example.org:3129" - proxy_os_https: "http://proxy.example.org:3128" - proxy_os_ftp: "http://proxy.example.org:3128" - proxy_os_noproxy: "127.0.0.1,localhost,192.168.1.0/24" - # Proxy to be used by YUM (/etc/yum.conf) - proxy_yum_url: "{{ proxy_os_http }}" + sriov: + network_interfaces: {eno1: 5} ``` + > NOTE: On various platform interfaces can have different name. For example `eth1` instead of `eno1`. Please verify interface name before deployment and do right changes. + +8. Set all necessary settings for `CERA 5G Central Office` in `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/all.yml`. -4. Build all docker images required and provide all necessary binaries. - - [dUPF](#dUPF) - - [UPF](#UPF) - - [AMF-SMF](#AMF-SMF) -5. Set all necessary settings for `CERA 5G NE` in `ido-converged-edge-experience-kits/host_vars/cera_5g_ne.yml`. - See [more details](#dUPF) for dUPF configuration - ```yaml - # Define PCI addresses (xxxx:xx:xx.x format) for i-upf - n3_pci_bus_address: "0000:3d:06.0" - n4_n9_pci_bus_address: "0000:3d:02.0" - n6_pci_bus_address: "0000:3d:02.1" - - # Define VPP VF interface names for i-upf - n3_vf_interface_name: "VirtualFunctionEthernet3d/6/0" - n4_n9_vf_interface_name: "VirtualFunctionEthernet3d/2/0" - n6_vf_interface_name: "VirtualFunctionEthernet3d/2/1" - ``` - ```yaml - # Define path where i-upf is located on remote host - upf_binaries_path: "/opt/flexcore-5g-rel/i-upf/" - ``` ```yaml - # PF interface name of N3 created VF - host_if_name_N3: "eno1" # PF interface name of N4, N6, N9 created VFs - host_if_name_N4_N6_n9: "eno2" - ``` - [OpenVino](#OpenVINO) settings if was set as active demo application - ```yaml - model: "pedestrian-detection-adas-0002" - display_host_ip: "" # update ip for visualizer HOST GUI. - save_video: "enable" - target_device: "CPU" + host_if_name_cn: "eno1" ``` -7. Set all necessary settings for `CERA 5G CN` in `ido-converged-edge-experience-kits/host_vars/cera_5g_cn.yml`. - For more details check: - - [UPF](#UPF) - - [AMF-SMF](#AMF-SMF) - ```yaml - # Define N4/N9 and N6 interface device PCI bus address - PCI_bus_address_N4_N9: '0000:3d:02.0' - PCI_bus_address_N6: '0000:3d:02.1' - # vpp interface name as per setup connection - vpp_interface_N4_N9_name: 'VirtualFunctionEthernet3d/2/0' - vpp_interface_N6_name: 'VirtualFunctionEthernet3d/2/1' - ``` - ```yaml - # 5gc binaries directory name - package_name_5gc: "5gc" - ``` +9. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/controller_group.yml` + + Provide names of `network interfaces` that are connected to the second server and number of VF's to be created. ```yaml - # psa-upf directory path - upf_binaries_path: '/opt/flexcore-5g-rel/psa-upf/' + sriov: + network_interfaces: {eno1: 5} ``` + +10. Edit file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/edgenode_group.yml` + + Set up CPUs isolation according to your hardware capabilities ```yaml - ## AMF-SMF vars + # Variables applied with the profile + tuned_vars: | + isolated_cores=1-16,25-40 + nohz=on + nohz_full=1-16,25-40 - # Define N2/N4 - PCI_bus_address_N2_N4: "0000:3d:02.3" + # CPUs to be isolated (for RT procesess) + cpu_isol: "1-16,25-40" + # CPUs not to be isolate (for non-RT processes) - minimum of two OS cores necessary for controller + cpu_os: "0,17-23,24,41-47" ``` - `CERA 5G CN` public ssh key + + Set up hugepages settings ```yaml - # Host public ssh key - host_ssh_key: "" + # Size of a single hugepage (2M or 1G) + hugepage_size: "1G" + # Amount of hugepages + hugepage_amount: "8" ``` - ```yaml - ## ConfigMap vars - # PF interface name of N3 created VF - host_if_name_N3: "eno2" - # PF interface name of N4, N6, N9 created VFs - host_if_name_N4_N6_n9: "eno1" - ``` -8. Provide correct IP for target servers in file `ido-converged-edge-experience-kits/cera_inventory.ini` - ```ini - [all] - cera_5g_ne ansible_ssh_user=root ansible_host=192.168.1.109 - cera_5g_cn ansible_ssh_user=root ansible_host=192.168.1.43 - ``` -9. Deploy CERA Experience Kit +11. Deploy Converged Edge Experience Kit (Near Edge and Central Office clusters simultaneously) + + Silent deployment: ```shell - ./deploy_cera.sh + python ./deploy.py ``` + > NOTE: In multicluster deployment logs are hidden by default. To check the logs `tail` tool can be used on deployment log files. + ## 5G Core Components This section describes in details how to build particular images and configure ansible for deployment. @@ -530,22 +442,17 @@ The `CERA dUPF` component is deployed on `CERA 5G Near Edge (cera_5g_ne)` node. #### Prerequisites -To deploy dUPF correctly it is needed to provide Docker image to Docker repository on target machine(cera_5g_ne). There is a script on the `open-ness/eddgeapps/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. - -#### Settings -Following variables need to be defined in `/host_vars/cera_5g_ne.yml` -```yaml -n3_pci_bus_address: "" - PCI bus address of VF, which is used for N3 interface by dUPF -n4_n9_pci_bus_address: "" - PCI bus address of VF, which is used for N4 and N9 interface by dUPF -n6_pci_bus_address: "" - PCI bus address of VF, which is used for N6 interface by dUPF +To deploy dUPF correctly it is needed to provide Docker image to Docker repository on target machine(cera_5g_ne). There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. -n3_vf_interface_name: "" - name of VF, which is used for N3 interface by dUPF -n4_n9_vf_interface_name: "" - name of VF, which is used for N4 and N9 interface by dUPF -n6_vf_interface_name: "" - name of VF, which is used for N6 interface by dUPF +```sh +./build_image.sh -b i-upf -i i-upf +``` -dpdk_driver_upf: "" - DPDK driver used (vfio-pci/igb_uio) to VFs bindings +#### Settings +Update interface name in file `ido-converged-edge-experience-kits/flavors/cera_5g_near_edge/all.yml` that is used for connection to Central Office cluster (AMF/SMF and UPF). -upf_binaries_path: "" - path where the dUPF binaries are located on the remote host +```yaml +host_if_name_cn: "eno1" ``` #### Configuration @@ -556,161 +463,52 @@ The dUPF is configured automatically during the deployment. The `User Plane Function (UPF)` is a part of 5G Core Network, it is responsible for packets routing. It has 2 separate interfaces for `N4/N9` and `N6` data lines. `N4/N9` interface is used for connection with `dUPF` and `AMF/SMF` (locally). `N6` interface is used for connection with `EDGE-APP`, `dUPF` and `Remote-DN` (locally). -The CERA UPF component is deployed on `CERA 5G Core Network (cera_5g_cn)` node. It is deployed as a POD - during deployment of OpenNESS with CERA 5G Near Edge flavor automatically. +The CERA UPF component is deployed on `CERA 5G Central Office (cera_5g_co)` node. It is deployed as a POD - during deployment of OpenNESS with CERA 5G Central Office flavor automatically. #### Deployment #### Prerequisites -To deploy UPF correctly it is needed to provide a Docker image to Docker Repository on target machine(cera_5g_ne and cera_5g_cn). There is a script on the `open-ness/eddgeapps/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. +To deploy UPF correctly it is needed to provide a Docker image to Docker Repository on target machine(cera_5g_ne and cera_5g_co). There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF` repo provided by CERA, which builds the image automatically. -#### Settings - -Following variables need to be defined in the `/host_vars/cera_5g_ne.yml` -```yaml -PCI_bus_address_N4_N9: "" - PCI bus address of VF, which is used for N4 and N9 interface by UPF -PCI_bus_address_N6: "" - PCI bus address of VF, which is used for N6 interface by UPF +```sh +./build_image.sh -b psa-upf -i psa-upf +``` -vpp_interface_N4_N9_name: "" - name of VF, which is used for N4 and N9 interface by UPF -vpp_interface_N6_name: "" - name of VF, which is used for N6 interface by UPF +#### Settings -dpdk_driver_upf: "" - DPDK driver used (vfio-pci/igb_uio) to VFs bindings +Update interface name in file `ido-converged-edge-experience-kits/flavors/cera_5g_central_office/all.yml` that is used for connection to Near Edge cluster (dUPF). -upf_binaries_path: "" - path where the UPF binaries are located on the remote host +```yaml +host_if_name_cn: "eno1" ``` #### Configuration -The UPF is configured automatically during the deployment. - +The `UPF` is configured automatically during the deployment. ### AMF-SMF #### Overview AMF-SMF is a part of 5G Core Architecture responsible for `Session Management(SMF)` and `Access and Mobility Management(AMF)` Functions - it establishes sessions and manages date plane packages. -The CERA `AMF-SMF` component is deployed on `CERA 5G Core Network (cera_5g_cn)` node and communicates with UPF and dUPF, so they must be deployed and configured before `AMF-SMF`. +The CERA `AMF-SMF` component is deployed on `CERA 5G Central Office (cera_5g_co)` node and communicates with UPF and dUPF, so they must be deployed and configured before `AMF-SMF`. -It is deployed in Virtual Machine with `Ubuntu 18.04 Cloud OS`, using `Kube-virt` on OpenNess platform - deploying OpenNess with CERA 5G Near Edge flavor automatically, configures and enables Kube-Virt plugin in OpenNess platform. +It is deployed as a POD - during deployment of OpenNESS with CERA 5G Central Office flavor automatically. #### Deployment #### Prerequisites -To deploy `AMF-SMF` correctly it is needed to provide image with `Ubuntu 18.04.1 Desktop (.img, .qcow2 format)` with required packages installed and directory with `AMF-SMF` binaries. +To deploy AMF-SMF correctly it is needed to provide Docker image to Docker repository on target machine(cera_5g_co). There is a script on the `https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/AMF_SMF` repo provided by CERA, which builds the image automatically. -#### Settings - -Following variables need to be defined in `/host_vars/cera_5g_cn.yml` -```yaml -PCI_bus_address_N2_N4: "" - PCI Bus address for VF (e.g. 0000:3a:01), which will be used for N2 and N4 interface by AMF-SMF (VF created from the same interface like Remote-DN and dUPF). - -host_ssh_key: "" - public ssh key of node - to generate public ssh key, please use on node command: ssh-keygen -t rsa and copy content of file located in $HOME/.ssh/id_rsa.pub (without ssh-rsa on beginning and user@hostname at the end) to variable. - -host_user_name: "" - username (e.g. root) of the node. - -And one variable in /group_vars/all.yml - -vm_image_path: "" - path where image of Virtual Machine (provided from script described above) is stored on host machine. +```sh +./build_image.sh -b amf-smf ``` +#### Settings +No special settings are required for AMF-SMF deployment. + #### Configuration -During the deployment, there is a Python script, which automatically configure `SMF` config files according to CERA setup. It changes IP subnet for `Local-DN` component in `AMF-SMF` configuration files. These settings can be changed manually if it is needed by User Setup. - -#### How to prepare image -Steps to do on host machine with CentOS - -1. Download Ubuntu 18.04.1 Desktop `.iso` image. - ```shell - wget http://old-releases.ubuntu.com/releases/18.04.1/ubuntu-18.04.1-desktop-amd64.iso - ``` -2. Check that `kvm_intel` is enabled in BIOS settings. - ```shell - dmesg | grep kvm -> should not display any disabled msg - lsmod | egrep 'kvm' - kvm_intel 183818 0 - kvm 624312 1 kvm_intel ->if BIOS VM enabled then kvm_intel should appear - irqbypass 13503 1 kvm - ``` -3. Enable VNC Server and install GNOME Desktop. - ```shell - yum groupinstall "GNOME Desktop" - yum install tigervnc-server xorg-x11-fonts-Type1 - vncserver -depth 24 -geometry 1920x1080 - ``` -4. Install Hypervisor packages and libraries. - ```shell - yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install virt-manager - systemctl start libvirtd - ``` -5. RUN `virt-manager` GUI application, select previous downloaded Ubuntu `.iso` image and the disk size (20GiB recommended) and follow the install process. After successful install take the next steps. -6. Change the grub file on Guest OS to allow console connection. - ```shell - vi /etc/default/grub - Add `console=ttyS0` to end of `GRUB_CMD_LINELINUX=` - ``` - Execute. - ```shell - grub-mkconfig -o /boot/grub/grub.cfg - Reboot board.. - ``` -7. Login to Guest OS using `virsh console`. - ```shell - virsh console
Edge WAN Overlay
| +| Overlay controller |is a Central Controller provides central control of SDEWAN overlay networks by automatically configuring the SDEWAN CNFs through SDEWAN CRD controller located in edge location clusters and hub clusters
| +| EWO Controller |To represent central overlay controller
| +| EWO Operator |To represent CRD controller
| +| EWO CNF |To represent OpenWRT based CNF.
| +| SDEWAN CRD Controller |is implemented as k8s CRD Controller, it manages CRDs (e.g. Firewall related CRDs, Mwan3 related CRDs and IPsec related CRDs etc.) and internally calls SDEWAN Restful API to do CNF configuration. And a remote client (e.g. SDEWAN Central Controller) can manage SDEWAN CNF configuration through creating/updating/deleting SDEWAN CRs.
| +| OpenWRT based CNF |The CNF is implemented based on OpenWRT, it enhances OpenWRT Luci web interface with SDEWAN controllers to provide Restful API for network functions configuration and control.
| diff --git a/doc/reference-architectures/core-network/openness_upf.md b/doc/reference-architectures/core-network/openness_upf.md index b8ea9c61..98a73b2d 100644 --- a/doc/reference-architectures/core-network/openness_upf.md +++ b/doc/reference-architectures/core-network/openness_upf.md @@ -45,7 +45,7 @@ As part of the end-to-end integration of the Edge cloud deployment using OpenNES # Purpose -This document provides the required steps to deploy UPF on the OpenNESS platform. 4G/(Long Term Evolution network)LTE or 5G UPF can run as network functions on the Edge node in a virtualized environment. The reference [Dockerfile](https://github.com/open-ness/edgeapps/blob/master/network-functions/core-network/5G/UPF/Dockerfile) and [5g-upf.yaml](https://github.com/open-ness/edgeapps/blob/master/network-functions/core-network/5G/UPF/5g-upf.yaml) provide details on how to deploy UPF as a Container Networking function (CNF) in a K8s pod on OpenNESS edge node using OpenNESS Enhanced Platform Awareness (EPA) features. +This document provides the required steps to deploy UPF on the OpenNESS platform. 4G/(Long Term Evolution network)LTE or 5G UPF can run as network functions on the Edge node in a virtualized environment. The reference [Dockerfile](https://github.com/open-ness/edgeapps/blob/master/network-functions/core-network/5G/UPF/Dockerfile) and [5g-upf.yaml](https://github.com/open-ness/edgeapps/blob/master/network-functions/core-network/5G/UPF/5g-upf.yaml) provide details on how to deploy UPF as a Cloud-native Network Function (CNF) in a K8s pod on OpenNESS edge node using OpenNESS Enhanced Platform Awareness (EPA) features. These scripts are validated through a reference UPF solution (implementation is based on Vector Packet Processing (VPP)) that is not part of the OpenNESS release. @@ -59,14 +59,17 @@ These scripts are validated through a reference UPF solution (implementation is 1. To keep the build and deploy process straightforward, the Docker\* build and image are stored on the Edge node. +2. Copy the upf binary package to the Docker build folder. Reference Docker files and the Helm chart for deploying the UPF is available at [edgeapps_upf_docker](https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF) and [edgeapps_upf_helmchart](https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/charts/upf) respectively + ```bash - ne-node# cd <5g-upf-binary-package> + ne-node# cp -rf <5g-upf-binary-package> edgeapps/network-functions/core-network/5G/UPF/upf ``` -2. Copy the Docker files to the node and build the Docker image. Reference Docker files and the Helm chart for deploying the UPF is available at [edgeapps_upf_docker](https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/5G/UPF) and [edgeapps_upf_helmchart](https://github.com/open-ness/edgeapps/tree/master/network-functions/core-network/charts/upf) respectively +3. Build the Docker image. ```bash - ne-node# ./build_image.sh + ne-node# cd edgeapps/network-functions/core-network/5G/UPF + ne-node# ./build_image.sh -b ./upf/ -i upf-cnf ne-node# docker image ls | grep upf upf-cnf 1.0 e0ce467c13d0 15 hours ago 490MB @@ -139,9 +142,9 @@ Below is a list of minimal configuration parameters for VPP-based applications s 3. Enable the vfio-pci/igb-uio driver on the node. The below example shows the enabling of the `igb_uio` driver: ```bash - ne-node# /opt/openness/dpdk-18.11.6/usertools/dpdk-devbind.py -b igb_uio 0000:af:0a.0 + ne-node# /opt/openness/dpdk-19.11.1/usertools/dpdk-devbind.py -b igb_uio 0000:af:0a.0 - ne-node# /opt/openness/dpdk-18.11.6/usertools/dpdk-devbind.py --status + ne-node# /opt/openness/dpdk-19.11.1/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:af:0a.0 'Ethernet Virtual Function 700 Series 154c' drv=igb_uio unused=i40evf,vfio-pci @@ -284,7 +287,7 @@ helm install \- **NOTE**: The command `groupadd vpp` needs to be given only for the first execution.
-
```bash
ne-controller# kubectl exec -it upf-cnf -- /bin/bash
- upf-cnf# groupadd vpp
- upf-cnf# ./run_upf.sh
+ upf-cnf# sudo -E ./run_upf.sh
```
## Uninstall UPF pod from OpenNESS controller
diff --git a/doc/reference-architectures/ran/openness_ran.md b/doc/reference-architectures/ran/openness_ran.md
index bd3bd9f5..0cac933d 100644
--- a/doc/reference-architectures/ran/openness_ran.md
+++ b/doc/reference-architectures/ran/openness_ran.md
@@ -8,39 +8,44 @@ Copyright (c) 2020 Intel Corporation
- [Building the FlexRAN image](#building-the-flexran-image)
- [FlexRAN hardware platform configuration](#flexran-hardware-platform-configuration)
- [BIOS](#bios)
+ - [Setting up CPU Uncore frequency](#setting-up-cpu-uncore-frequency)
- [Host kernel command line](#host-kernel-command-line)
+- [Deploying Access Edge CERA for FlexRAN](#deploying-access-edge-cera-for-flexran)
- [Deploying and Running the FlexRAN pod](#deploying-and-running-the-flexran-pod)
- [Setting up 1588 - PTP based Time synchronization](#setting-up-1588---ptp-based-time-synchronization)
- [Setting up PTP](#setting-up-ptp)
- [Primary clock](#primary-clock)
- [Secondary clock](#secondary-clock)
- [BIOS configuration](#bios-configuration)
+- [CPU frequency configuration](#cpu-frequency-configuration)
- [References](#references)
# Introduction
-Radio Access Network (RAN) is the edge of wireless network. 4G and 5G base stations form the key network function for the edge deployment. In OpenNESS, FlexRAN is used as a reference for 4G and 5G base stations as well as 4G and 5G end-to-end testing.
-FlexRAN offers high-density baseband pooling that could run on a distributed Telco\* cloud to provide a smart indoor coverage solution and next-generation fronthaul architecture. This 4G and 5G platform provides the open platform ‘smarts’ for both connectivity and new applications at the edge of the network, along with the developer tools to create these new services. FlexRAN running on the Telco Cloud provides low latency compute, storage, and network offload from the edge. Thus, saving network bandwidth.
+Radio Access Network (RAN) is the edge of wireless network. 4G and 5G base stations form the key network function for the edge deployment. In OpenNESS, FlexRAN is used as a reference for 4G and 5G base stations as well as 4G and 5G end-to-end testing.
-FlexRAN 5GNR Reference PHY is a baseband PHY Reference Design for a 4G and 5G base station, using Intel® Xeon® processor family with Intel® architecture. This 5GNR Reference PHY consists of a library of c-callable functions that are validated on several technologies from Intel (Intel® microarchitecture code name Broadwell, Intel® microarchitecture code name Skylake, Cascade Lake, and Ice Lake) and demonstrates the capabilities of the software running different 5GNR L1 features. The functionality of these library functions is defined by the relevant sections in [3GPP TS 38.211, 212, 213, 214, and 215]. Performance of the Intel 5GNR Reference PHY meets the requirements defined by the base station conformance tests in [3GPP TS 38.141]. This library of functions will be used by Intel partners and end customers as a foundation for their product development. Reference PHY is integrated with third-party L2 and L3 to complete the base station pipeline.
+FlexRAN offers high-density baseband pooling that could run on a distributed Telco\* cloud to provide a smart indoor coverage solution and next-generation fronthaul architecture. This 4G and 5G platform provides the open platform ‘smarts’ for both connectivity and new applications at the edge of the network, along with the developer tools to create these new services. FlexRAN running on the Telco Cloud provides low latency compute, storage, and network offload from the edge. Thus, saving network bandwidth.
-The diagram below shows FlexRAN DU (Real-time L1 and L2) deployed on the OpenNESS platform with the necessary microservices and Kubernetes\* enhancements required for real-time workload deployment.
+FlexRAN 5GNR Reference PHY is a baseband PHY Reference Design for a 4G and 5G base station, using Intel® Xeon® processor family with Intel® architecture. This 5GNR Reference PHY consists of a library of c-callable functions that are validated on several technologies from Intel (Intel® microarchitecture code name Broadwell, Intel® microarchitectures code name Skylake, Cascade Lake, and Intel® microarchitecture Ice Lake) and demonstrates the capabilities of the software running different 5GNR L1 features. The functionality of these library functions is defined by the relevant sections in [3GPP TS 38.211, 212, 213, 214, and 215]. Performance of the Intel 5GNR Reference PHY meets the requirements defined by the base station conformance tests in [3GPP TS 38.141]. This library of functions will be used by Intel partners and end customers as a foundation for their product development. Reference PHY is integrated with third-party L2 and L3 to complete the base station pipeline.
+
+The diagram below shows FlexRAN DU (Real-time L1 and L2) deployed on the OpenNESS platform with the necessary microservices and Kubernetes\* enhancements required for real-time workload deployment.
-This document aims to provide the steps involved in deploying FlexRAN 5G (gNb) on the OpenNESS platform.
+This document aims to provide the steps involved in deploying FlexRAN 5G (gNb) on the OpenNESS platform.
->**NOTE**: This document covers both FlexRAN 4G and 5G. All the steps mentioned in this document use 5G for reference. Refer to the [FlexRAN 4G Reference Solution L1 User Guide #570228](https://cdrdv2.intel.com/v1/dl/getContent/570228) for minor updates needed to build, deploy, and test FlexRAN 4G.
+>**NOTE**: This document covers both FlexRAN 4G and 5G. All the steps mentioned in this document use 5G for reference. Refer to the [FlexRAN 4G Reference Solution L1 User Guide #570228](https://cdrdv2.intel.com/v1/dl/getContent/570228) for minor updates needed to build, deploy, and test FlexRAN 4G.
-# Building the FlexRAN image
+# Building the FlexRAN image
This section explains the steps involved in building the FlexRAN image. Only L1 and L2-stub will be part of these steps. Real-time L2 (MAC and RLC) and non-real-time L2 and L3 are out of scope as it is a part of the third-party component.
1. Contact your Intel representative to obtain the package
2. Untar the FlexRAN package.
3. Set the required environmental variables:
- ```
- export RTE_SDK=$localPath/dpdk-19.11
+
+ ```shell
+ export RTE_SDK=$localPath/dpdk-20.11
export RTE_TARGET=x86_64-native-linuxapp-icc
export WIRELESS_SDK_TARGET_ISA=avx512
export RPE_DIR=${flexranPath}/libs/ferrybridge
@@ -56,97 +61,204 @@ This section explains the steps involved in building the FlexRAN image. Only L1
export FLEXRAN_SDK=${DIR_WIRELESS_SDK}/install
export DIR_WIRELESS_TABLE_5G=${flexranPath}/bin/nr5g/gnb/l1/table
```
- >**NOTE**: The environmental variables path must be updated according to your installation and file/directory names.
-4. Build L1, WLS interface between L1, L2, and L2-Stub (testmac):
- `./flexran_build.sh -r 5gnr_sub6 -m testmac -m wls -m l1app -b -c`
+
+ >**NOTE**: The environmental variables path must be updated according to your installation and file/directory names.
+
+4. Build L1, WLS interface between L1, L2, and L2-Stub (testmac):
+
+ ```shell
+ # ./flexran_build.sh -r 5gnr_sub6 -m testmac -m wls -m l1app -b -c
+ ```
+
5. Once the build has completed, copy the required binary files to the folder where the Docker\* image is built. This can be done by using a provided example [build-du-dev-image.sh](https://github.com/open-ness/edgeapps/blob/master/network-functions/ran/5G/du-dev/build-du-dev-image.sh) script from Edge Apps OpenNESS repository, it will copy the files from the paths provided as environmental variables in previous step. The script will copy the files into the right directory containing the Dockerfile and commence the docker build.
+
```shell
- git clone https://github.com/open-ness/edgeapps.git
- cd edgeapps/network-functions/ran/5G/du-dev
- ./build-du-dev-image.sh
+ # git clone https://github.com/open-ness/edgeapps.git
+ # cd edgeapps/network-functions/ran/5G/du-dev
+ # ./build-du-dev-image.sh
```
- The list of binary files that are used is documented in [dockerfile](https://github.com/open-ness/edgeapps/blob/master/network-functions/ran/5G/flexRAN-gnb/Dockerfile)
- - ICC, IPP mpi and mkl Runtime
- - DPDK build target directory
- - FlexRAN test vectors (optional)
- - FlexRAN L1 and testmac (L2-stub) binary
- - FlexRAN SDK modules
- - FlexRAN WLS share library
- - FlexRAN CPA libraries
+ The list of binary files that are used is documented in [dockerfile](https://github.com/open-ness/edgeapps/blob/master/network-functions/ran/5G/du-dev/Dockerfile)
+ - ICC, IPP mpi and mkl Runtime
+ - DPDK build target directory
+ - FlexRAN test vectors (optional)
+ - FlexRAN L1 and testmac (L2-stub) binary
+ - FlexRAN SDK modules
+ - FlexRAN WLS share library
+ - FlexRAN CPA libraries
6. The following example reflects the Docker image [expected by Helm chart](https://github.com/open-ness/edgeapps/blob/master/network-functions/ran/charts/du-dev/values.yaml), user needs to adjust the IP address and port of the Harbor registry where Docker image will be pushed:
-
+
```shell
image:
repository: