[TASK] Refactor internal handling of container image definitions (#605)

ARTIFACT_INVENTORY.md

@@ -16,22 +16,23 @@ registry/repository/image_name:version
 
 | Subsystem| Component | Fully Qualified Container-Image Name (registry/repository/image_name:version)|
 |----|----|----|
-| Logging | BusyBox (OpenSearch) | docker.io/library/busybox|
-| Logging | Fluent Bit | cr.fluentbit.io/fluent/fluent-bit:2.1.10|
-| Logging | Elasticsearch Exporter | quay.io/prometheuscommunity/elasticsearch-exporter:v1.6.0|
-| Logging | OpenSearch | docker.io/opensearchproject/opensearch:2.10.0|
-| Logging | OpenSearch Dashboards| docker.io/opensearchproject/opensearch-dashboards:2.10.0|
-| Metrics | Alertmanager | quay.io/prometheus/alertmanager:v0.26.0|
-| Metrics | Ghostunnel | docker.io/ghostunnel/ghostunnel:v1.7.1|
-| Metrics | Grafana | docker.io/grafana/grafana:10.2.1|
-| Metrics | Admission Webhook | registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6|
-| Metrics | Kube State Metrics | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0|
-| Metrics | Node Exporter | quay.io/prometheus/node-exporter:v1.7.0|
-| Metrics | Prometheus | quay.io/prometheus/prometheus:v2.47.1|
-| Metrics | Prometheus Operator | quay.io/prometheus-operator/prometheus-operator:v0.69.1|
-| Metrics | Configuration Reloader (Alertmanager, Prometheus) | quay.io/prometheus-operator/prometheus-config-reloader:v0.69.1|
-| Metrics | Prometheus Pushgateway | quay.io/prom/pushgateway:v1.6.2|
-| Metrics | Auto-load Sidecars (Grafana) | quay.io/kiwigrid/k8s-sidecar:1.25.2|
+| Logging | BusyBox (OpenSearch) | docker.io/library/busybox:latest |
+| Logging | Fluent Bit | cr.fluentbit.io/fluent/fluent-bit:2.1.10 |
+| Logging | Elasticsearch Exporter | quay.io/prometheuscommunity/elasticsearch-exporter:v1.6.0 |
+| Logging | OpenSearch | docker.io/opensearchproject/opensearch:2.10.0 |
+| Logging | OpenSearch Dashboards| docker.io/opensearchproject/opensearch-dashboards:2.10.0 |
+| Metrics | Alertmanager | quay.io/prometheus/alertmanager:v0.26.0 |
+| Metrics | Grafana | docker.io/grafana/grafana:10.2.1 |
+| Metrics | Admission Webhook | registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6 |
+| Metrics | Kube State Metrics | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0 |
+| Metrics | Node Exporter | quay.io/prometheus/node-exporter:v1.7.0 |
+| Metrics | Prometheus | quay.io/prometheus/prometheus:v2.47.1 |
+| Metrics | Prometheus Operator | quay.io/prometheus-operator/prometheus-operator:v0.69.1 |
+| Metrics | Configuration Reloader (Alertmanager, Prometheus) | quay.io/prometheus-operator/prometheus-config-reloader:v0.69.1 |
+| Metrics | Prometheus Pushgateway | quay.io/prometheus/pushgateway:v1.6.2 |
+| Metrics | Auto-load Sidecars (Grafana) | quay.io/kiwigrid/k8s-sidecar:1.25.2 |
+| Metrics | OpenShift OAUTH Proxy (Grafana, OpenShift only) | registry.redhat.io/openshift4/ose-oauth-proxy:latest |
+| Metrics | Tempo | docker.io/grafana/tempo:2.2.0 |
 
 ## Table 2. Helm Chart Repositories
 This table identifies the Helm repositories that contain the Helm charts used by SAS Viya Monitoring for Kubernetes.
@@ -49,10 +50,11 @@ This table identifies the Helm charts used by SAS Viya Monitoring for Kubernetes
 
 | Subsystem | Component | Helm Chart Repository | Helm Chart Name |Helm Chart Version | Helm Archive File Name|
 |--|--|--|--|--|--|
-| Logging | Elasticsearch Exporter| prometheus-community| prometheus-elasticsearch-exporter| 5.3.1| prometheus-elasticsearch-exporter-5.3.1.tgz
-| Logging | Fluent Bit| fluent| fluent-bit| 0.40.0| fluent-bit-0.40.0.tgz
-| Logging | OpenSearch| opensearch| opensearch| 2.15.0| opensearch-2.15.0.tgz
-| Logging | OpenSearch Dashboard| opensearch| opensearch-dashboards| 2.13.0| opensearch-dashboards-2.13.0.tgz
-| Metrics | Grafana (on OpenShift)| grafana| grafana| 7.0.4| grafana-7.0.4.tgz
-| Metrics | Kube Prometheus Stack| prometheus-community| kube-prometheus-stack| 54.0.1| kube-prometheus-stack-54.0.1.tgz
-| Metrics | Prometheus Pushgateway| prometheus-community| prometheus-pushgateway| 2.4.2| prometheus-pushgateway-2.4.2.tgz
+| Logging | Elasticsearch Exporter| prometheus-community | prometheus-elasticsearch-exporter | 5.3.1 | prometheus-community/prometheus-elasticsearch-exporter-5.3.1.tgz |
+| Logging | Fluent Bit| fluent | fluent-bit | 0.40.0 | fluent/fluent-bit-0.40.0.tgz |
+| Logging | OpenSearch| opensearch | opensearch | 2.15.0 | opensearch/opensearch-2.15.0.tgz |
+| Logging | OpenSearch Dashboard| opensearch | opensearch-dashboards | 2.13.0 | opensearch/opensearch-dashboards-2.13.0.tgz |
+| Metrics | Grafana (on OpenShift)| grafana | grafana | 7.0.4 | grafana/grafana-7.0.4.tgz |
+| Metrics | Kube Prometheus Stack| prometheus-community | kube-prometheus-stack | 54.0.1 | prometheus-community/kube-prometheus-stack-54.0.1.tgz |
+| Metrics | Prometheus Pushgateway| prometheus-community | prometheus-pushgateway | 2.4.2 | prometheus-community/prometheus-pushgateway-2.4.2.tgz |
+| Metrics | Tempo | grafana | tempo | 1.5.0 | grafana/tempo-1.5.0.tgz |

CHANGELOG.md

@@ -1,11 +1,19 @@
 # SAS Viya Monitoring for Kubernetes
 
-
 ## Unreleased
+* **Overall**
+  * [TASK] Refactored how container image and Helm chart version information is handled to permit automatically generating this information from files.  Note
+that this change does NOT alter how users provide this information should they wish to change it.  User should continue to include this information in the 
+appropriate user values yaml file within their USER_DIR directory.  However, specifying a Helm chart or container image version different than the default
+should rarely be necessary or appropriate.
 
 * **Metrics**
   * [FIX] Replaced obsolete container image name for OpenShift oauth proxy container
 
+* **Logging**
+  * [REMOVAL] The deploy_eventrouter.sh script has been removed.  The [Event Router component](https://github.com/vmware-archive/eventrouter) it deployed
+is no longer actively developed and was replaced with a Fluent Bit deployment focused on collecting Kubernetes events in our 1.2.19 release.
+
 ## Version 1.2.21 (17JAN2024)
 
 * **Metrics**

bin/airgap-include.sh

@@ -1,4 +1,4 @@
-# Copyright © 2023, SAS Institute Inc., Cary, NC, USA.  All Rights Reserved.
+# Copyright © 2023-2024, SAS Institute Inc., Cary, NC, USA.  All Rights Reserved.
 # SPDX-License-Identifier: Apache-2.0
 
 # This file is not marked as executable as it is intended to be sourced
@@ -12,7 +12,16 @@ if [ "$AIRGAP_SOURCED" == "" ]; then
         log_error "Please provide the URL for the private image registry and try again"
         exit 1
     else
+
         AIRGAP_IMAGE_PULL_SECRET_NAME=${AIRGAP_IMAGE_PULL_SECRET_NAME:-"v4m-image-pull-secret"}
+
+        # Check for the image pull secret for the air gap environment
+        if [ -z "$(kubectl get secret -n $V4M_NS $AIRGAP_IMAGE_PULL_SECRET_NAME -o name --ignore-not-found)" ]; then
+           log_error "The image pull secret, [$AIRGAP_IMAGE_PULL_SECRET_NAME], was not detected"
+           log_error "Please add the image pull secret to the [$V4M_NS] namespace and run the deployment script again"
+           exit 1
+        fi
+
         AIRGAP_HELM_REPO=${AIRGAP_HELM_REPO:-"$AIRGAP_REGISTRY"}
         AIRGAP_HELM_FORMAT=${AIRGAP_HELM_FORMAT:-"oci"}
 
@@ -35,35 +44,3 @@ if [ "$AIRGAP_SOURCED" == "" ]; then
     export AIRGAP_SOURCED=true
 fi
 
-## The user will need to create the namespace and secret before running the deployment scripts.
-## This function will produce an error if the secret is not found in the environment.
-function checkForAirgapSecretInNamespace {
-    secretName="$1"
-    namespace="$2" 
-    if [ -z "$(kubectl get secret -n $namespace | grep $secretName)" ]; then
-        log_error "The image pull secret, [$secretName], was not detected"
-        log_error "Please add the image pull secret to the [$namespace] namespace and run the deployment script again"
-        exit 1
-    fi
-}
-
-function replaceAirgapValuesInFiles {
-    fileToUpdate=$1
-    filename="$(echo $fileToUpdate | sed -n -e 's/^.*airgap\///p')"
-
-    updatedAirgapValuesFile="$airgapDir/$filename"
-
-    cp $fileToUpdate $updatedAirgapValuesFile
-
-    log_debug "Replacing air gap placeholders for [$updatedAirgapValuesFile]"
-    if echo "$OSTYPE" | grep 'darwin' > /dev/null 2>&1; then
-      sed -i '' "s/__AIRGAP_REGISTRY__/$AIRGAP_REGISTRY/g" $updatedAirgapValuesFile
-      sed -i '' "s/__AIRGAP_IMAGE_PULL_SECRET_NAME__/$AIRGAP_IMAGE_PULL_SECRET_NAME/g" $updatedAirgapValuesFile
-    else
-      sed -i "s/__AIRGAP_REGISTRY__/$AIRGAP_REGISTRY/g" $updatedAirgapValuesFile
-      sed -i "s/__AIRGAP_IMAGE_PULL_SECRET_NAME__/$AIRGAP_IMAGE_PULL_SECRET_NAME/g" $updatedAirgapValuesFile
-    fi
-}
-
-export -f checkForAirgapSecretInNamespace
-export -f replaceAirgapValuesInFiles

bin/common.sh

@@ -61,7 +61,10 @@ if [ "$SAS_COMMON_SOURCED" = "" ]; then
         log_debug "No component_versions.env file found"
     fi
 
-
+    if [ "$V4M_OMIT_IMAGE_KEYS" == "true" ]; then
+       log_warn "******This feature is NOT intended for use outside the project maintainers*******"
+       log_warn "Environment variable V4M_OMIT_IMAGE_KEYS set to [true]; container image information from component_versions.env will be ignored."
+    fi
 
     export USER_DIR=${USER_DIR:-$(pwd)}
     if [ -d "$USER_DIR" ]; then
@@ -82,7 +85,7 @@ if [ "$SAS_COMMON_SOURCED" = "" ]; then
     log_info "User directory: $USER_DIR"
 
     export AIRGAP_DEPLOYMENT=${AIRGAP_DEPLOYMENT:-false}
-    
+
     CHECK_HELM=${CHECK_HELM:-true}
     if [ "$CHECK_HELM" == "true" ]; then
        source bin/helm-include.sh
@@ -253,3 +256,101 @@ export -f trap_add
 export -f errexit_msg
 export -f disable_sa_token_automount
 export -f enable_pod_token_automount
+
+function parseFullImage {
+   fullImage="$1"
+   unset REGISTRY REPOS IMAGE VERSION FULL_IMAGE_ESCAPED
+
+   if [[ "$1" =~ (.*)\/(.*)\/(.*)\:(.*) ]]; then
+
+      REGISTRY="${BASH_REMATCH[1]}"
+      REPOS="${BASH_REMATCH[2]}"
+      IMAGE="${BASH_REMATCH[3]}"
+      VERSION="${BASH_REMATCH[4]}"
+      FULL_IMAGE_ESCAPED="$REGISTRY\/$REPOS\/$IMAGE\:$VERSION"
+      return 0
+   else
+      log_warn "Invalid value for full container image; does not fit expected pattern [$1]."
+      return 1
+   fi
+}
+
+
+function v4m_replace {
+
+    if echo "$OSTYPE" | grep 'darwin' > /dev/null 2>&1; then
+      sed -i '' s/"$1"/"$2"/g  "$3"
+    else
+      sed -i  s/"$1"/"$2"/g  "$3"
+    fi
+}
+
+function generateImageKeysFile {
+
+   #arg1 Full container image
+   #arg2 name of template file
+   #arg3 prefix to insert in placeholders (optional)
+
+   local pullsecret_text
+
+   if ! parseFullImage "$1";  then
+      log_error "Unable to parse full image [$1]"
+      return 1
+   fi
+
+   prefix=${3:-""}
+
+   imageKeysFile="$TMP_DIR/imageKeysFile.yaml"
+   template_file=$2
+
+   if [ "$template_file" != "$imageKeysFile" ]; then
+      rm -f  $imageKeysFile
+      cp $template_file  $imageKeysFile
+   else
+      log_debug "Modifying an existing imageKeysFile"
+   fi
+
+   if [ "$V4M_OMIT_IMAGE_KEYS" == "true" ]; then
+      cp $TMP_DIR/empty.yaml $imageKeysFile
+      return 0
+   fi
+
+   if [ "$AIRGAP_DEPLOYMENT" == "true" ]; then
+      GLOBAL_REGISTRY_OSBUG="$AIRGAP_REGISTRY"
+      GLOBAL_REGISTRY="$AIRGAP_REGISTRY"
+      REGISTRY="$AIRGAP_REGISTRY"
+
+      if [ -n "$AIRGAP_IMAGE_PULL_SECRET_NAME" ]; then
+         pullsecrets_text="[name: ""$AIRGAP_IMAGE_PULL_SECRET_NAME""]"
+         pullsecret_text="$AIRGAP_IMAGE_PULL_SECRET_NAME"
+      else
+         pullsecrets_text="[]"
+         pullsecret_text="null"
+      fi
+   else
+      GLOBAL_REGISTRY_OSBUG='""'
+      GLOBAL_REGISTRY="null"
+      pullsecrets_text="[]"
+      pullsecret_text="null"
+   fi
+
+   v4m_pullPolicy=${V4M_PULL_POLICY:-"IfNotPresent"}
+
+   v4m_replace "__${prefix}GLOBAL_REGISTRY_OSBUG__"    "$GLOBAL_REGISTRY_OSBUG"          "$imageKeysFile"
+   v4m_replace "__${prefix}GLOBAL_REGISTRY__"    "$GLOBAL_REGISTRY"          "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_REGISTRY__"     "$REGISTRY"                 "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_REPO_3LEVEL__"  "$REGISTRY\/$REPOS\/$IMAGE" "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_REPO_2LEVEL__"  "$REPOS\/$IMAGE"            "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE__"              "$IMAGE"                    "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_TAG__"          "$VERSION"                  "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_PULL_POLICY__"  "$v4m_pullPolicy"           "$imageKeysFile"
+   v4m_replace "__${prefix}IMAGE_PULL_SECRET__"  "$pullsecret_text"          "$imageKeysFile"       #Handle Charts Accepting a Single Image Pull Secret
+   v4m_replace "__${prefix}IMAGE_PULL_SECRETS__" "$pullsecrets_text"         "$imageKeysFile"       #Handle Charts Accepting Multiple Image Pull Secrets
+
+   return 0
+}
+
+
+export -f parseFullImage
+export -f v4m_replace
+export -f generateImageKeysFile

bin/helm-include.sh

@@ -24,6 +24,11 @@ if [ "$HELM_VER_MAJOR" == "2" ]; then
     exit 1
 fi
 
+if [ "$V4M_HELM_USE_LATEST"  == "true" ]; then
+  log_warn "******This feature is NOT intended for use outside the project maintainers*******"
+  log_warn "Environment variable V4M_HELM_USE_LATEST set; deploying *latest* version of all Helm charts"
+fi
+
 function helm2ReleaseExists {
   release=$1
   log_debug "Checking for Helm 2.x release of [$release]"
@@ -119,10 +124,19 @@ function get_helmchart_reference {
       echo "${chart_repository}/${chart_name}"
   fi
 }
+function get_helm_versionstring {
+    if [ "$V4M_HELM_USE_LATEST"  == "true" ]; then
+       :  # return null string
+    else
+       echo "--version $1"
+    fi
 
+    return
+}
 export HELM_VER_FULL HELM_VER_MAJOR HELM_VER_MINOR HELM_VER_PATCH
 export -f helm2ReleaseExists
 export -f helm3ReleaseExists
 export -f helm2ReleaseCheck
 export -f helmRepoAdd
 export -f get_helmchart_reference
+export -f get_helm_versionstring

component_versions.env

@@ -12,36 +12,54 @@
 ESEXPORTER_HELM_CHART_REPO=prometheus-community
 ESEXPORTER_HELM_CHART_NAME=prometheus-elasticsearch-exporter
 ESEXPORTER_HELM_CHART_VERSION=5.3.1
+ES_EXPORTER_FULL_IMAGE="quay.io/prometheuscommunity/elasticsearch-exporter:v1.6.0"
 
 #Fluent Bit
 FLUENTBIT_HELM_CHART_REPO=fluent
 FLUENTBIT_HELM_CHART_NAME=fluent-bit
 FLUENTBIT_HELM_CHART_VERSION=0.40.0
+FB_FULL_IMAGE="cr.fluentbit.io/fluent/fluent-bit:2.1.10"
 
 #OpenSearch
 OPENSEARCH_HELM_CHART_REPO=opensearch
 OPENSEARCH_HELM_CHART_NAME=opensearch
 OPENSEARCH_HELM_CHART_VERSION=2.15.0
+OS_FULL_IMAGE="docker.io/opensearchproject/opensearch:2.10.0"
+OS_SYSCTL_FULL_IMAGE="docker.io/library/busybox:latest"
 
 #OpenSearch Dashboards
 OSD_HELM_CHART_REPO=opensearch
 OSD_HELM_CHART_NAME=opensearch-dashboards
 OSD_HELM_CHART_VERSION=2.13.0
+OSD_FULL_IMAGE="docker.io/opensearchproject/opensearch-dashboards:2.10.0"
 
 #Grafana (when deployed on OpenShift)
 OPENSHIFT_GRAFANA_CHART_REPO=grafana
 OPENSHIFT_GRAFANA_CHART_NAME=grafana
 OPENSHIFT_GRAFANA_CHART_VERSION=7.0.4
+OPENSHIFT_OAUTHPROXY_FULL_IMAGE="registry.redhat.io/openshift4/ose-oauth-proxy:latest"
+
+#Grafana (everywhere)
+GRAFANA_FULL_IMAGE="docker.io/grafana/grafana:10.2.1"
+GRAFANA_SIDECAR_FULL_IMAGE="quay.io/kiwigrid/k8s-sidecar:1.25.2"
 
 #Kube-Prometheus Stack
 KUBE_PROM_STACK_CHART_REPO=prometheus-community
 KUBE_PROM_STACK_CHART_NAME=kube-prometheus-stack
 KUBE_PROM_STACK_CHART_VERSION=54.0.1
+ALERTMANAGER_FULL_IMAGE="quay.io/prometheus/alertmanager:v0.26.0"
+ADMWEBHOOK_FULL_IMAGE="registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20221220-controller-v1.5.1-58-g787ea74b6"
+KSM_FULL_IMAGE="registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0"
+NODEXPORT_FULL_IMAGE="quay.io/prometheus/node-exporter:v1.7.0"
+PROMETHEUS_FULL_IMAGE="quay.io/prometheus/prometheus:v2.47.1"
+PROMOP_FULL_IMAGE="quay.io/prometheus-operator/prometheus-operator:v0.69.1"
+CONFIGRELOAD_FULL_IMAGE="quay.io/prometheus-operator/prometheus-config-reloader:v0.69.1" 
 
 #Pushgateway
 PUSHGATEWAY_CHART_REPO=prometheus-community
 PUSHGATEWAY_CHART_NAME=prometheus-pushgateway
 PUSHGATEWAY_CHART_VERSION=2.4.2
+PUSHGATEWAY_FULL_IMAGE="quay.io/prometheus/pushgateway:v1.6.2"
 
 #Prometheus Operator CRD
 PROM_OPERATOR_CRD_VERSION=v0.69.1
@@ -50,4 +68,4 @@ PROM_OPERATOR_CRD_VERSION=v0.69.1
 TEMPO_CHART_REPO=grafana
 TEMPO_CHART_NAME=tempo
 TEMPO_CHART_VERSION=1.5.0
-
+TEMPO_FULL_IMAGE="docker.io/grafana/tempo:2.2.0"

logging/bin/common.sh

@@ -62,7 +62,13 @@ if [ "$SAS_LOGGING_COMMON_SOURCED" = "" ]; then
        export LOG_XSRF_HEADER="kbn-xsrf: true"
     fi
 
+
     export V4M_NS=$LOG_NS
+
+    if [ "$AIRGAP_DEPLOYMENT" == "true" ]; then
+       source bin/airgap-include.sh
+    fi
+
     source bin/version-include.sh
 
     export SAS_LOGGING_COMMON_SOURCED=true