Dind storage r1 (#109)

* dind storage from master -draft1

* venona storage -1

* fixed node selector for runtime

* fixed node selector for runtime

* fixed uninstall

* storage to README

* single storage class

* storage before merge 1

* storage before merge 1

Co-authored-by: kosta709 <[email protected]>

Author: kosta709

README.md

@@ -114,6 +114,88 @@ rules:
   * Bind your user with cluster-admin kubernetes clusterrole
     > `kubectl create clusterrolebinding NAME --clusterrole cluster-admin --user YOUR_USER`
 
+#### Pipeline Storage with docker cache support
+
+##### **GKE LocalSSD**
+*Prerequisite:* [GKE custer with local SSD](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd)
+
+*Install venona for using GKE Local SSD:*
+```
+codefresh install runtime [options] \
+                    --set-value=Storage.LocalVolumeParentDir=/mnt/disks/ssd0/codefresh-volumes 
+                    --kube-selector=cloud.google.com/gke-local-ssd=true
+```
+
+##### **GCE Disks** 
+*Prerequisite:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of google disks
+There are 3 options to provide cloud credentials on GCE:
+* run venona dind-volume-provisioniner on node with iam role which is allowed to create/delete/get of google disks
+* create Google Service Account with ComputeEngine.StorageAdmin, download its key and pass it to venona installed with `--set-file=Storage.GooogleServiceAccount=/path/to/google-service-account.json`
+* use [Google Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) to assign iam role to `volume-provisioner-venona` service account 
+
+*Note*: Builds will be running in single availability zone, so you must to specify AvailabilityZone params
+
+
+*Install venona for using GKE Disks:*
+```
+codefresh install runtime [options] \
+                    --set-value=Storage.Backend=gcedisk \
+                    --set-value=Storage.AvailabilityZone=us-central1-a \
+                    --kube-node-selector=failure-domain.beta.kubernetes.io/zone=us-central1-a \
+                    [--set-file=Storage.GoogleServiceAccount=/path/to/google-service-account.json]
+```
+
+##### **Amazon EBS**
+
+*Prerequisite:* volume provisioner (dind-volume-provisioner) should have permissions to create/delete/get of aws ebs
+Minimal iam policy for dind-volume-provisioner:
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "ec2:AttachVolume",
+        "ec2:CreateSnapshot",
+        "ec2:CreateTags",
+        "ec2:CreateVolume",
+        "ec2:DeleteSnapshot",
+        "ec2:DeleteTags",
+        "ec2:DeleteVolume",
+        "ec2:DescribeInstances",
+        "ec2:DescribeSnapshots",
+        "ec2:DescribeTags",
+        "ec2:DescribeVolumes",
+        "ec2:DetachVolume"
+      ],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+There are 3 options to provide cloud credentials on AWS:
+* run venona dind-volume-provisioniner on node with the iam role - use `--set-value Storage.VolumeProvisioner.NodeSelector=node-label=value` option
+* create AWS IAM User, assign it the permissions above and suppy aws credentials to venona installer `--set-value=Storage.AwsAccessKeyId=ABCDF --set-value=Storage.AwsSecretAccessKey=ZYXWV`
+
+* use [Aws Identity for Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to assign iam role to `volume-provisioner-venona` service account
+
+*Notes*: 
+- Builds will be running in single availability zone, so you must specify AvailabilityZone parameter `--set-value=Storage.AvailabilityZone=<aws-az>` and build-node-selector `--build-node-selector=failure-domain.beta.kubernetes.io/zone=<aws-az>` in case of multizone cluster
+
+- We support both [in-tree ebs](https://kubernetes.io/docs/concepts/storage/volumes/#awselasticblockstore) (`--set-value=Storage.Backend=ebs`) volumes and ebs-csi(https://github.com/kubernetes-sigs/aws-ebs-csi-driver) (`--set-value=Storage.Backend=ebs-csi`)
+
+*Install Command to run pipelines on ebs volumes*
+```
+codefresh install runtime [options] \
+                    --set-value=Storage.Backend=ebs \
+                    --set-value=Storage.AvailabilityZone=us-east-1d \
+                    --kube-node-selector=failure-domain.beta.kubernetes.io/zone=us-east-1d \
+                    [--set-value Storage.VolumeProvisioner.NodeSelector=kubernetes.io/role=master] \
+                    [--set-value Storage.AwsAccessKeyId=ABCDF --set-value Storage.AwsSecretAccessKey=ZYXWV]
+```
+
 #### Kubernetes RBAC
 Installation of Venona on Kubernetes cluster installing 2 groups of objects,
 Each one has own RBAC needs and therefore, created roles(and cluster-roles)

package.json

@@ -1,6 +1,6 @@
 {
   "name": "venona",
-  "version": "1.0.7",
+  "version": "1.0.8",
   "description": "Codefresh agent to run on Codefresh's runtime environment and execute pipeline",
   "main": "index.js",
   "scripts": {

venonactl/VERSION

@@ -1 +1 @@
-1.0.7
+1.0.8

venonactl/cmd/cmdutils.go

@@ -23,6 +23,8 @@ import (
 	"gopkg.in/yaml.v2"
 	k8sApi "k8s.io/api/core/v1"
 	"k8s.io/client-go/tools/clientcmd"
+
+	"helm.sh/helm/v3/pkg/strvals"
 )
 
 var (
@@ -230,7 +232,55 @@ func extendStoreWithAgentAPI(logger logger.Logger, token string, agentID string)
 func (ns nodeSelector) String() string {
 	var s string
 	for k, v := range ns {
-		s = fmt.Sprintf("%s: %s", k, v)
+		s = fmt.Sprintf("%s: %q", k, v)
 	}
 	return s
 }
+
+// Parsing helpers --set-value , --set-file 
+// by https://github.com/helm/helm/blob/ec1d1a3d3eb672232f896f9d3b3d0797e4f519e3/pkg/cli/values/options.go#L41
+
+// parses --set-value options 
+func parseSetValues(setValuesOpts []string) (map[string]interface{}, error) {
+	base := map[string]interface{}{}
+	for _, value := range setValuesOpts {
+		if err := strvals.ParseInto(value, base); err != nil {
+			return nil, fmt.Errorf("Cannot parse option --set-value %s", value)
+		}
+	}
+	return base, nil
+}
+
+// parses --set-file options 
+func parseSetFiles(setFilesOpts []string) (map[string]interface{}, error) {
+	base := map[string]interface{}{}
+	for _, value := range setFilesOpts {
+		reader := func(rs []rune) (interface{}, error) {
+			bytes, err := ioutil.ReadFile(string(rs))
+			return string(bytes), err
+		}
+		if err := strvals.ParseIntoFile(value, base, reader); err != nil {
+			return nil, fmt.Errorf("Cannot parse option --set-file %s", value)
+		}
+	}
+	return base, nil
+}
+
+func mergeMaps(a, b map[string]interface{}) map[string]interface{} {
+	out := make(map[string]interface{}, len(a))
+	for k, v := range a {
+		out[k] = v
+	}
+	for k, v := range b {
+		if v, ok := v.(map[string]interface{}); ok {
+			if bv, ok := out[k]; ok {
+				if bv, ok := bv.(map[string]interface{}); ok {
+					out[k] = mergeMaps(bv, v)
+					continue
+				}
+			}
+		}
+		out[k] = v
+	}
+	return out
+}

venonactl/cmd/install-runtime.go

@@ -32,11 +32,14 @@ var installRuntimeCmdOptions struct {
 		namespace string
 		inCluster bool
 		context   string
+		nodeSelector string
 	}
 	storageClass           string
 	runtimeEnvironmentName string
 	kubernetesRunnerType   bool
 	tolerations            string
+	templateValues         []string
+	templateFileValues     []string
 }
 
 var installRuntimeCmd = &cobra.Command{
@@ -78,6 +81,12 @@ var installRuntimeCmd = &cobra.Command{
 			s.KubernetesAPI.Tolerations = tolerations
 		}
 
+		kns, err := parseNodeSelector(installRuntimeCmdOptions.kube.nodeSelector)
+		if err != nil {
+			dieOnError(err)
+		}
+		s.KubernetesAPI.NodeSelector = kns.String()
+
 		builder := plugins.NewBuilder(lgr)
 		isDefault := isUsingDefaultStorageClass(installRuntimeCmdOptions.storageClass)
 
@@ -118,8 +127,24 @@ var installRuntimeCmd = &cobra.Command{
 		}
 
 		builderInstallOpt.KubeBuilder = getKubeClientBuilder(s.KubernetesAPI.ContextName, s.KubernetesAPI.Namespace, s.KubernetesAPI.ConfigPath, s.KubernetesAPI.InCluster)
-		var err error
 		values := s.BuildValues()
+
+		if len(installRuntimeCmdOptions.templateValues) > 0 {
+			setValues, err := parseSetValues(installRuntimeCmdOptions.templateValues)
+			if err != nil {
+				dieOnError(err)
+			}
+			values = mergeMaps(values, setValues)
+		}
+
+		if len(installRuntimeCmdOptions.templateFileValues) > 0 {
+			setFileValues, err := parseSetFiles(installRuntimeCmdOptions.templateFileValues)
+			if err != nil {
+				dieOnError(err)
+			}
+			values = mergeMaps(values, setFileValues)
+		}
+
 		for _, p := range builder.Get() {
 			values, err = p.Install(builderInstallOpt, values)
 			if err != nil {
@@ -146,6 +171,10 @@ func init() {
 	installRuntimeCmd.Flags().BoolVar(&installRuntimeCmdOptions.kube.inCluster, "in-cluster", false, "Set flag if venona is been installed from inside a cluster")
 	installRuntimeCmd.Flags().BoolVar(&installRuntimeCmdOptions.dryRun, "dry-run", false, "Set to true to simulate installation")
 	installRuntimeCmd.Flags().BoolVar(&installRuntimeCmdOptions.kubernetesRunnerType, "kubernetes-runner-type", false, "Set the runner type to kubernetes (alpha feature)")
+	installRuntimeCmd.Flags().StringVar(&installRuntimeCmdOptions.kube.nodeSelector, "kube-node-selector", "", "The kubernetes node selector \"key=value\" to be used by venona resources (default is no node selector)")
 	installRuntimeCmd.Flags().StringVar(&installRuntimeCmdOptions.tolerations, "tolerations", "", "The kubernetes tolerations as JSON string to be used by venona resources (default is no tolerations)")
 
+	installRuntimeCmd.Flags().StringArrayVar(&installRuntimeCmdOptions.templateValues, "set-value", []string{}, "Set values for templates, example: --set-value LocalVolumesDir=/mnt/disks/ssd0/codefresh-volumes")
+	installRuntimeCmd.Flags().StringArrayVar(&installRuntimeCmdOptions.templateFileValues, "set-file", []string{}, "Set values for templates from file, example: --set-file Storage.GoogleServiceAccount=/path/to/service-account.json")
+
 }

venonactl/cmd/uninstall-runtime.go

@@ -38,7 +38,6 @@ var uninstallRuntimeCmd = &cobra.Command{
 
 		s.CodefreshAPI = &store.CodefreshAPI{}
 		s.AgentAPI = &store.AgentAPI{}
-
 
 		builder := plugins.NewBuilder(lgr)
 		if uninstallRunimeCmdOptions.kube.context == "" {
@@ -47,7 +46,8 @@ var uninstallRuntimeCmd = &cobra.Command{
 		if uninstallRunimeCmdOptions.kube.namespace == "" {
 			dieOnError(fmt.Errorf("Namespace name is required to in order to uninstall agent"))
 		}
-
+		s.KubernetesAPI.ContextName = uninstallRunimeCmdOptions.kube.context
+		s.KubernetesAPI.Namespace = uninstallRunimeCmdOptions.kube.namespace
 
 		if uninstallRunimeCmdOptions.kubeVenona.kubePath == "" {
 			uninstallRunimeCmdOptions.kubeVenona.kubePath = kubeConfigPath

venonactl/go.mod

@@ -1,46 +1,27 @@
 module github.com/codefresh-io/venona/venonactl
 
+go 1.13
+
 require (
-	contrib.go.opencensus.io/exporter/ocagent v0.4.3 // indirect
-	github.com/Azure/go-autorest v11.4.0+incompatible // indirect
-	github.com/Masterminds/goutils v1.1.0 // indirect
+	github.com/Azure/go-autorest/autorest v0.10.0 // indirect
 	github.com/Masterminds/semver v1.5.0 // indirect
 	github.com/Masterminds/sprig v2.22.0+incompatible
-	github.com/codefresh-io/go-sdk v0.17.0
+	github.com/codefresh-io/go-sdk v0.18.0
 	github.com/dustin/go-humanize v1.0.0
-	github.com/google/go-github/v21 v21.0.0
-	github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367 // indirect
-	github.com/google/uuid v1.1.1 // indirect
-	github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d // indirect
-	github.com/gophercloud/gophercloud v0.0.0-20190206021053-df38e1611dbe // indirect
-	github.com/gregjones/httpcache v0.0.0-20170728041850-787624de3eb7 // indirect
-	github.com/hashicorp/go-version v1.1.0
+	github.com/gophercloud/gophercloud v0.8.0 // indirect
 	github.com/huandu/xstrings v1.3.0 // indirect
-	github.com/imdario/mergo v0.3.5 // indirect
-	github.com/inconshreveable/log15 v0.0.0-20180818164646-67afb5ed74ec
-	github.com/json-iterator/go v0.0.0-20180701071628-ab8a2e0c74be // indirect
-	github.com/mattn/go-colorable v0.1.0 // indirect
-	github.com/mattn/go-isatty v0.0.4 // indirect
-	github.com/mitchellh/copystructure v1.0.0 // indirect
-	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/imdario/mergo v0.3.8 // indirect
+	github.com/inconshreveable/log15 v0.0.0-20200109203555-b30bc20e4fd1
+	github.com/mattn/go-colorable v0.1.6 // indirect
 	github.com/olekukonko/tablewriter v0.0.4
-	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
-	github.com/spf13/cobra v0.0.5
-	github.com/spf13/viper v1.6.1
-	go.opencensus.io v0.19.0 // indirect
-	google.golang.org/appengine v1.4.0 // indirect
-	gopkg.in/inf.v0 v0.9.0 // indirect
-	gopkg.in/yaml.v2 v2.2.7
-	k8s.io/api v0.0.0-20181221193117-173ce66c1e39
-	k8s.io/apimachinery v0.0.0-20181127025237-2b1284ed4c93
-	k8s.io/client-go v10.0.0+incompatible
-	k8s.io/klog v0.0.0-20181108234604-8139d8cb77af // indirect
-	sigs.k8s.io/yaml v1.1.0 // indirect
+	github.com/spf13/cobra v0.0.6
+	github.com/spf13/viper v1.6.2
+	gopkg.in/yaml.v2 v2.2.8
+	helm.sh/helm/v3 v3.1.1
+	k8s.io/api v0.17.3
+	k8s.io/apimachinery v0.17.3
+	k8s.io/client-go v0.17.3
+	k8s.io/utils v0.0.0-20200229041039-0a110f9eb7ab // indirect
 )
 
-replace git.apache.org/thrift.git => github.com/apache/thrift v0.0.0-20181218151757-9b75e4fe745a
-
-replace github.com/census-instrumentation/opencensus-proto => github.com/census-instrumentation/opencensus-proto v0.0.3-0.20181214143942-ba49f56771b8
-
-go 1.13
+replace github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.2+incompatible