tests: Scripts for e2e tests (kubeflow#2128)

* remove old test files

Signed-off-by: Kimonas Sotirchos <[email protected]>

* gitignore: Don't track pyc files

Signed-off-by: Kimonas Sotirchos <[email protected]>

* flake8: Introduce linting file

Signed-off-by: Kimonas Sotirchos <[email protected]>

* hack: Introduce scripts for cluster manipulation

Signed-off-by: Kimonas Sotirchos <[email protected]>

* tests: Add e2e test

Signed-off-by: Kimonas Sotirchos <[email protected]>

* GH action for running e2e test

Signed-off-by: Kimonas Sotirchos <[email protected]>

* Reduce the installed components and system reqs

Signed-off-by: Kimonas Sotirchos <[email protected]>

* kserve: Add simple kustomization file

To avoid having to use --load_restrictor none we'll need to wrap the
KServe manifests inside a kustomization.yaml file.

Signed-off-by: Kimonas Sotirchos <[email protected]>

* unittests: Fix unit tests

Signed-off-by: Kimonas Sotirchos <[email protected]>

* gh: Remove action for e2e tests

We should use prow instead to trigger our e2e tests.

Signed-off-by: Kimonas Sotirchos <[email protected]>

.flake8

@@ -0,0 +1,4 @@
+[flake8]
+docstring_convention = google
+exclude = assets,__init__.py,__pycache__
+ignore = D100,D103,D104,D107,W503

.github/workflows/manifests_unittests.yaml

@@ -13,9 +13,14 @@ jobs:
     - name: Check out repo
       uses: actions/checkout@v2
 
-    - name: Unit Test
+    - name: Install Kustomize
+      working-directory: ./tests/e2e
       run: |
-        cd tests
-        make test
+        curl -Lo ./kustomize https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
+        chmod +x ./kustomize
+        sudo mv kustomize /usr/local/bin
 
+    - name: Unit Test
+      run: |
+        kustomize build example
 

.gitignore

@@ -2,6 +2,8 @@
 .vscode
 .DS_Store
 
+**/*.pyc
+
 # Swap
 [._]*.s[a-v][a-z]
 !*.svg  # comment out if you don't need vector files

contrib/kserve/upstream/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- kserve.yaml
+# For KF 1.5 we are including both KFServing and KServe. Thus we install the
+# standalone kserve manifests, to avoid conflicts with 0.6.1 KFServing.
+#- kserve_kubeflow.yaml

example/kustomization.yaml

@@ -53,5 +53,4 @@ resources:
 - ../common/user-namespace/base
 
 # KServe
-#- ../contrib/kserve/upstream/kserve_kubeflow.yaml
-- ../contrib/kserve/upstream/kserve.yaml
+- ../contrib/kserve/upstream

hack/kind-cluster-1-20.yaml

@@ -0,0 +1,23 @@
+# https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492
+# https://github.com/istio/istio/blob/e02690fbfb8bda564582b27d22d9e8e6e00422a5/prow/config/trustworthy-jwt.yaml#L1-L13
+# This configs KinD to spin up a k8s cluster with trustworthy jwt (Service Account Token Volume Projection) feature.
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: manifests-1-20
+nodes:
+- role: control-plane
+  image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
+- role: worker
+  image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
+- role: worker
+  image: kindest/node:1.20.7@sha256:688fba5ce6b825be62a7c7fe1415b35da2bdfbb5a69227c499ea4cc0008661ca
+kubeadmConfigPatches:
+  - |
+    apiVersion: kubeadm.k8s.io/v1beta2
+    kind: ClusterConfiguration
+    metadata:
+      name: config
+    apiServer:
+      extraArgs:
+        "service-account-issuer": "kubernetes.default.svc"
+        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

hack/kind-cluster-1-21.yaml

@@ -0,0 +1,23 @@
+# https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492
+# https://github.com/istio/istio/blob/e02690fbfb8bda564582b27d22d9e8e6e00422a5/prow/config/trustworthy-jwt.yaml#L1-L13
+# This configs KinD to spin up a k8s cluster with trustworthy jwt (Service Account Token Volume Projection) feature.
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: manifests-1-21
+nodes:
+- role: control-plane
+  image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
+- role: worker
+  image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
+- role: worker
+  image: kindest/node:1.21.2@sha256:19c2315068fd5951aa478ef7b9d1771572c8ea58fbfbf7bc81f7b153679d7a6c
+kubeadmConfigPatches:
+  - |
+    apiVersion: kubeadm.k8s.io/v1beta2
+    kind: ClusterConfiguration
+    metadata:
+      name: config
+    apiServer:
+      extraArgs:
+        "service-account-issuer": "kubernetes.default.svc"
+        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

hack/setup-kubeflow-light.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+
+# Copyright 2021 The Kubeflow Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This shell script is used to setup Katib deployment.
+set -euo pipefail
+
+TIMEOUT=600s  # 10mins
+
+echo "Creating Kubeflow namespace..."
+kubectl create namespace kubeflow
+
+echo "Deploying Cert-Manager."
+kustomize build common/cert-manager/cert-manager/base | kubectl apply -f -
+
+echo "Waiting for Cert Manager pods to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n cert-manager --all --for=condition=Ready pod
+
+echo "Deploying Istio."
+kustomize build common/istio-1-11/istio-crds/base | kubectl apply -f -
+kustomize build common/istio-1-11/istio-namespace/base | kubectl apply -f -
+kustomize build common/istio-1-11/istio-install/base | kubectl apply -f -
+
+echo "Waiting for istio-system Pods to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n istio-system --all --for=condition=Ready pod
+
+echo "Deploying Knative."
+function install_knative {
+    kustomize build common/knative/knative-serving/overlays/gateways | kubectl apply -f -
+}
+
+while ! install_knative;
+do
+    echo "Retrying to install knative..."
+    sleep 10
+done
+
+kustomize build common/knative/knative-eventing/base | kubectl apply -f -
+kustomize build common/istio-1-11/cluster-local-gateway/base | kubectl apply -f -
+
+echo "Waiting for knative-serving Pods to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n knative-serving --all --for=condition=Ready pod
+
+echo "Deploying KFP."
+function install_kfp {
+    kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user | kubectl apply -f - --validate=false
+}
+
+while ! install_kfp;
+do
+    echo "Retrying to install kfp..."
+    sleep 10
+done
+
+echo "Waiting for kubeflow/ml-pipelines to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=ml-pipeline --for=condition=Ready pod
+
+echo "Deploying KFServing."
+kustomize build apps/kfserving/upstream/overlays/kubeflow | kubectl apply -f -
+
+echo "Waiting for kubeflow/kfserving to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=kfserving --for=condition=Ready pod
+
+echo "Deploying Katib."
+kustomize build apps/katib/upstream/installs/katib-with-kubeflow | kubectl apply -f -
+
+echo "Waiting for kubeflow/katib to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l katib.kubeflow.org/component=controller --for=condition=Ready pod
+
+echo "Deploying Training Operator."
+kustomize build apps/training-operator/upstream/overlays/kubeflow | kubectl apply -f -
+
+echo "Waiting for kubeflow/training-operator to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l control-plane=kubeflow-training-operator --for=condition=Ready pod
+
+echo "Installing Profiles Controller."
+kustomize build apps/profiles/upstream/overlays/kubeflow | kubectl apply -f -
+
+echo "Waiting for kubeflow/profiles-controller to become ready..."
+sleep 5
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l kustomize.component=profiles --for=condition=Ready pod
+
+echo "Creating user resources."
+kustomize build common/user-namespace/base | kubectl apply -f -
+kustomize build common/cert-manager/kubeflow-issuer/base | kubectl apply -f -
+

hack/setup-kubeflow.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+# Copyright 2021 The Kubeflow Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This shell script is used to setup Katib deployment.
+set -euo pipefail
+
+TIMEOUT=600s  # 10mins
+
+echo "Creating Kubeflow namespace..."
+kubectl create namespace kubeflow
+
+echo "Deploying all Kubeflow components..."
+function install_kubeflow {
+    kustomize build example --load_restrictor none | kubectl apply -f - --validate=false
+}
+
+while ! install_kubeflow;
+do
+    echo "Retrying to apply resources"
+    sleep 10
+done
+
+echo "---"
+echo "Waiting for all Kubeflow components to become ready."
+
+echo "Waiting for Cert Manager pods to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n cert-manager --all --for=condition=Ready pod
+
+echo "Waiting for istio-system Pods to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n istio-system --all --for=condition=Ready pod
+
+echo "Waiting for knative-serving Pods to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n knative-serving --all --for=condition=Ready pod
+
+echo "Waiting for kubeflow/ml-pipelines to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=ml-pipeline --for=condition=Ready pod
+
+echo "Waiting for kubeflow/kfserving to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l app=kfserving --for=condition=Ready pod
+
+echo "Waiting for kubeflow/katib to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l katib.kubeflow.org/component=controller --for=condition=Ready pod
+
+echo "Waiting for kubeflow/training-operator to become ready..."
+kubectl wait --timeout=${TIMEOUT} -n kubeflow -l control-plane=kubeflow-training-operator --for=condition=Ready pod

tests/README.md

@@ -1,30 +1,25 @@
 # Kustomize Manifest Tests
 
-## Unittests
+## E2E
 
-There are two goals for the unittests:
+There are 2 goals for the e2e tests:
+1. Verify that core components can be applied and become Ready
+2. Virify that core CRDs can be created and succeed
 
-1. Verify that kustomizations are applied correctly (kubeflow/manifests#1014)
-1. Verify that various conventions are enforced (kubeflow/manifests#1015)
+### Components
 
-### Verifying Kustomizations Are Applied Correctly
+The e2e are installing everything by using the [single-install
+command](../README.md#install-with-a-single-command). This means that all of
+the core and common (Istio, Knativ, Cert Manager) are being installed and
+included in the tests.
 
-Examples of kustomizations that we would like to verify are applied and generate the expected output
+### Test Suite
 
-* Patches
-* Variable substitution
-* Composition of resources
+The e2e tests are completely independent of the underlying K8s cluster, as well
+as the platform of the cluster. These tests should be able to run in real
+world clusters, as well as ephemeral ones like KinD.
 
-The general approach to doing this is
-
-1. Check in one more "kustomization.yaml" files corresponding to test cases
-1. Run "kustomize build -o ..." and check in the output as the expected test output
-
-   * Reviewers can verify changes to the expected output to ensure changes have the desired effect on the expected output
-1. Unittests run "kustomize build" and compare output to expected output to ensure kustomize packages are in sync with the expected output
-1. Make commands make it easy to regenerate the expected output as part of a change.
-
-   ```
-   cd tests
-   make generate-changed-only
-   ```
+The tests are also explicitly bypassing any authentication system of the
+installation. The goal of these tests are to ensure that a common use case that
+deploys all of Kubeflow's components can succeed. Thus we only want to test
+that the core CRDs can be successfully applied and complete.

tests/e2e/.gitignore

@@ -0,0 +1 @@
+**/pids.env