Consume Quarks Links as environment variables

[#170725085](https://www.pivotaltracker.com/story/show/170725085)
Using Quarks links (entanglements) on the consumer side required changes
in the operator so that a link property can be used through an
environment variable and as a file with just the property content. In
theory, this should enable consumers to retrieve the property value in
their respective project with minimal adjustments.

Introduce flattened link properties in the Kubernetes secrets that
contain the link properties. The name and the data of the link secret
change from `link-deploymentname-name`
```
"nats.nats": "{\"nats\": { \"port\": 4442}, ... }"
```
to a new flatten look and name `link-deploymentname-type-name`
```
"nats.port": "4442"
"nats.user": "admin"
```
The link type and name become part of the secret name and will not be
used in the secret data as the root key anymore. Therefore, all code
sections and tests that wait or rely on the old link secret name were
updated to use the new names. This relies on new convenience functions
to create the name for both the link secret name and the type/name pair.

The `cmd_instance_group_resolver` code and tests were updated to write
the flattened properties in the `provides.json`, which is later used by
a Quarks Job to persist the data into the link secrets.

The `job_factory` code and tests were updated to use the new "fan-out"
style of the Quarks Job project that writes the content of the
aforementioned `provides.json` into separate link secrets, each only
containing one key/value pair. With other words, there will be one
secret per each link type/name tuple.

The `pod_mutator` code and tests were changed to not mount the link
properties as one file (`link.yaml`) anymore, but that each entry in the
link properties becomes its own file in the target container. Also, each
entry in the properties will result in an environment variable starting
with the prefix `LINK_` and the respective value. For example, the nats
username will be exposed as `LINK_NATS_USER` in the containers.

Update documentation to include the changes with respect to the newly added
environment variables that are based on the link secrets.

bin/include/dependencies

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-git_sha="6a177e8"
+git_sha="62fcf84"
 quarks_job_release="v0.0.0-0.g$git_sha"
 
 # QUARKS_JOB_IMAGE_TAG is used for integration tests

docs/entanglements.md

@@ -22,7 +22,7 @@ We construct link information like so:
 
 > If multiple secrets or services are found with the same link information, the operator should error
 
-### Example
+### Example (Native -> BOSH)
 
 ```yaml
 kind: Secret
@@ -37,7 +37,7 @@ spec:
 
 Using this secret, I should be able to use `link("nats").p("password")` in one of my BOSH templates.
 
-```
+```yaml
 apiVersion: v1
 kind: Service
 metadata:
@@ -66,20 +66,35 @@ If the service is changed, or the list of pods selected by the service is change
 
 In this case, the BOSH component is a provider, and the native component is a consumer.
 
-The operator creates link Secrets for all providers in a BOSH deployment.
+The operator creates link secrets for all providers in a BOSH deployment. Each secret contains a flattened map with the provided properties:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: link-test-nats-nats
+data:
+  nats.password: YXBwYXJlbnRseSwgeW91Cg==
+  nats.port: aGF2ZSB0b28K
+  nats.user: bXVjaCB0aW1lCg==
+```
 
 If a pod is annotated with the following:
-  - `quarks.cloudfoundry.org/deployment: foo`
-  - `quarks.cloudfoundry.org/consumes: '[{"name":"nats","type":"nats"}]'`
-The operator will:
-  - mutate the pod and mount the secret as `/quarks/link/DEPLOYMENT/<name>/link.yaml`
 
-Where `<name>` is the name of the link, e.g. 'nats'.
+- `quarks.cloudfoundry.org/deployment: foo`
+- `quarks.cloudfoundry.org/consumes: '[{"name":"nats","type":"nats"}]'`
+
+The operator will mutate the pod to:
+
+- mount the link secrets as `/quarks/link/DEPLOYMENT/<type>.<name>/<key>`
+- add an environment variable for each key in the secret data mapping: `LINK_<key>`
+
+The `<name>` and `<type>` are the respective link type and name, for example the nats release uses `nats` for both the name and the type of the link. Whereas the `<key>` describes the BOSH properties style flattened property key with its dot-style, for example `nats.password`. The key name is modified to be all upper case and without dots in the context of an environment variable, therefore `nats.password` becomes `LINK_NATS_PASSWORD` in the container.
 
 If link information changes, the operator will trigger an update (restart) of the deployment or statefulset owning the pod.
 This can be done by updating the template of the pod using an annotation.
 
-## Example
+### Example (BOSH -> Native)
 
 an Eirini Helm Chart
 
@@ -94,6 +109,7 @@ The OPI process of Eirini required the NATS password and IP.
     spec:
 
 ```
+
 and a CF-Deployment with Operator
 Instance Groups:
 
@@ -102,4 +118,3 @@ Instance Groups:
 - Gorouter
 - NATS
   provides: nats
-

e2e/kube/bosh_link_entangled_pods_test.go

@@ -11,18 +11,15 @@ import (
 )
 
 var _ = Describe("BOSHLinkEntanglements", func() {
-	const jobProperties = `{"nats":{"password":"onetwothreefour","port":4222,"user":"admin"}}`
-	const changedProperties = `{"nats":{"password":"qwerty1234","port":4222,"user":"admin"}}`
-
 	apply := func(p string) error {
 		yamlPath := path.Join(examplesDir, p)
 		return cmdHelper.Apply(namespace, yamlPath)
 	}
 
-	checkEntanglement := func(podName, expect string) error {
+	checkEntanglement := func(podName, cmd, expect string) error {
 		return kubectl.RunCommandWithCheckString(
 			namespace, podName,
-			"cat /quarks/link/nats-deployment/nats/link.yaml",
+			cmd,
 			expect,
 		)
 	}
@@ -44,7 +41,7 @@ var _ = Describe("BOSHLinkEntanglements", func() {
 
 	Context("when creating a bosh deployment", func() {
 		It("creates secrets for a all BOSH links", func() {
-			exist, err := kubectl.SecretExists(namespace, "link-nats-deployment-nats")
+			exist, err := kubectl.SecretExists(namespace, "link-nats-deployment-nats-nats")
 			Expect(err).ToNot(HaveOccurred())
 			Expect(exist).To(BeTrue())
 		})
@@ -61,8 +58,9 @@ var _ = Describe("BOSHLinkEntanglements", func() {
 				err := apply("quarks-link/entangled-sts.yaml")
 				Expect(err).ToNot(HaveOccurred())
 				podWait(selector)
-				err = checkEntanglement(podName, jobProperties)
-				Expect(err).ToNot(HaveOccurred())
+
+				Expect(checkEntanglement(podName, "cat /quarks/link/nats-deployment/nats-nats/nats.password", "onetwothreefour")).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "echo $LINK_NATS_USER", "admin")).ToNot(HaveOccurred())
 			})
 
 			By("restarting pods when the link secret changes", func() {
@@ -76,8 +74,8 @@ var _ = Describe("BOSHLinkEntanglements", func() {
 				Expect(err).ToNot(HaveOccurred(), "waiting for restart annotation on entangled pod")
 				podWait(selector)
 
-				err = checkEntanglement(podName, changedProperties)
-				Expect(err).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "cat /quarks/link/nats-deployment/nats-nats/nats.password", "qwerty1234")).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "echo $LINK_NATS_USER", "admin")).ToNot(HaveOccurred())
 			})
 		})
 	})
@@ -97,8 +95,8 @@ var _ = Describe("BOSHLinkEntanglements", func() {
 				podName = getPodName(selector)
 				podWait("pod/" + podName)
 
-				err = checkEntanglement(podName, jobProperties)
-				Expect(err).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "cat /quarks/link/nats-deployment/nats-nats/nats.password", "onetwothreefour")).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "echo $LINK_NATS_USER", "admin")).ToNot(HaveOccurred())
 			})
 
 			By("restarting pods when the link secret changes", func() {
@@ -119,8 +117,8 @@ var _ = Describe("BOSHLinkEntanglements", func() {
 				)
 				Expect(err).ToNot(HaveOccurred(), "waiting for restart annotation on entangled pod")
 
-				err = checkEntanglement(podName, changedProperties)
-				Expect(err).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "cat /quarks/link/nats-deployment/nats-nats/nats.password", "qwerty1234")).ToNot(HaveOccurred())
+				Expect(checkEntanglement(podName, "echo $LINK_NATS_USER", "admin")).ToNot(HaveOccurred())
 			})
 		})
 	})

go.mod

@@ -1,8 +1,8 @@
 module code.cloudfoundry.org/cf-operator
 
 require (
-	code.cloudfoundry.org/quarks-job v0.0.0-20200128080450-0deec9592a1e
-	code.cloudfoundry.org/quarks-utils v0.0.0-20200128080244-7f8de3f1673c
+	code.cloudfoundry.org/quarks-job v0.0.0-20200130141800-25a764f89866
+	code.cloudfoundry.org/quarks-utils v0.0.0-20200130110052-eac67a73088b
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bmatcuk/doublestar v1.1.1 // indirect
 	github.com/charlievieth/fs v0.0.0-20170613215519-7dc373669fa1 // indirect

go.sum

@@ -3,10 +3,14 @@ cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMT
 cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
 code.cloudfoundry.org/quarks-job v0.0.0-20200128080450-0deec9592a1e h1:eKxuDhITX02+q4+NagYT/eakaDk9IMNUgGt6Dlno2bk=
 code.cloudfoundry.org/quarks-job v0.0.0-20200128080450-0deec9592a1e/go.mod h1:fLlgnrSanve71EtkVswD8hoI6U2RYM4+0S3A2ajvTEI=
+code.cloudfoundry.org/quarks-job v0.0.0-20200130141800-25a764f89866 h1:EU2BCSJjcQLpqvkcJ8ShXS+LUb3OTrsmHJqeLR9bwP8=
+code.cloudfoundry.org/quarks-job v0.0.0-20200130141800-25a764f89866/go.mod h1:fLlgnrSanve71EtkVswD8hoI6U2RYM4+0S3A2ajvTEI=
 code.cloudfoundry.org/quarks-utils v0.0.0-20200127150718-47028dacbc7c h1:nEbvnRc7JUgzLFfopuvdPBhT4nHQulxg8iVVPxuVZOM=
 code.cloudfoundry.org/quarks-utils v0.0.0-20200127150718-47028dacbc7c/go.mod h1:d2OaSM1qVE/7Zo1imovL7CZCOAShFePFMI3jlpMcp14=
 code.cloudfoundry.org/quarks-utils v0.0.0-20200128080244-7f8de3f1673c h1:m0VovyZz1Ny2OtGn3siS6Q7JumekLW0hvIwoeKadl4c=
 code.cloudfoundry.org/quarks-utils v0.0.0-20200128080244-7f8de3f1673c/go.mod h1:d2OaSM1qVE/7Zo1imovL7CZCOAShFePFMI3jlpMcp14=
+code.cloudfoundry.org/quarks-utils v0.0.0-20200130110052-eac67a73088b h1:vngvL8ncDz/ihUgnNZplK1gCwnxDppucvBLeYZKFqVc=
+code.cloudfoundry.org/quarks-utils v0.0.0-20200130110052-eac67a73088b/go.mod h1:d2OaSM1qVE/7Zo1imovL7CZCOAShFePFMI3jlpMcp14=
 github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
 github.com/Azure/go-autorest/autorest v0.9.0/go.mod h1:xyHB1BMZT0cuDHU7I0+g046+BFDTQ8rEZB0s4Yfa6bI=
 github.com/Azure/go-autorest/autorest/adal v0.5.0/go.mod h1:8Z9fGy2MpX0PvDjB1pEgQTmVqjGhiHBW7RJJEciWzS0=

integration/quarks_link_entangled_pod_test.go

@@ -29,16 +29,6 @@ var _ = Describe("Entangled Pods PodMutator", func() {
 		return names
 	}
 
-	volumeKeyToPaths := func(volumes []corev1.Volume) []string {
-		keys := make([]string, len(volumes))
-		for i, v := range volumes {
-			if len(v.Secret.Items) > 0 {
-				keys[i] = v.Secret.Items[0].Key
-			}
-		}
-		return keys
-	}
-
 	volumeMountNames := func(mounts []corev1.VolumeMount) []string {
 		names := make([]string, len(mounts))
 		for i, m := range mounts {
@@ -87,12 +77,11 @@ var _ = Describe("Entangled Pods PodMutator", func() {
 				Expect(err).NotTo(HaveOccurred())
 
 				Expect(p.Spec.Volumes).To(HaveLen(2))
-				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats"))
-				Expect(volumeKeyToPaths(p.Spec.Volumes)).To(ContainElement("nats.nats"))
+				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats-nats"))
 
 				for _, c := range p.Spec.Containers {
 					Expect(c.VolumeMounts).To(HaveLen(2))
-					Expect(volumeMountNames(c.VolumeMounts)).To(ContainElement("link-nats-deployment-nats"))
+					Expect(volumeMountNames(c.VolumeMounts)).To(ContainElement("link-nats-deployment-nats-nats"))
 				}
 			})
 		})
@@ -105,10 +94,15 @@ var _ = Describe("Entangled Pods PodMutator", func() {
 			tearDowns = append(tearDowns, tearDown)
 
 			otherSecret := env.QuarksLinkSecret(
-				deploymentName, "ig",
-				"type", "name",
-				`{"foo":[1,2,3],{"password":"abc"}}`,
+				deploymentName,
+				"type",
+				"name",
+				map[string][]byte{
+					"foo":      []byte("[1,2,3]"),
+					"password": []byte("abc"),
+				},
 			)
+
 			tearDown, err = env.CreateSecret(env.Namespace, otherSecret)
 			Expect(err).NotTo(HaveOccurred())
 			tearDowns = append(tearDowns, tearDown)
@@ -125,15 +119,14 @@ var _ = Describe("Entangled Pods PodMutator", func() {
 				Expect(err).NotTo(HaveOccurred())
 
 				Expect(p.Spec.Volumes).To(HaveLen(3))
-				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats"))
-				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-ig"))
-				Expect(volumeKeyToPaths(p.Spec.Volumes)).To(ContainElement("type.name"))
+				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats-nats"))
+				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-type-name"))
 
 				for _, c := range p.Spec.Containers {
 					Expect(c.VolumeMounts).To(HaveLen(3))
 					mounts := c.VolumeMounts
-					Expect(volumeMountNames(mounts)).To(ContainElement("link-nats-deployment-nats"))
-					Expect(volumeMountNames(mounts)).To(ContainElement("link-nats-deployment-ig"))
+					Expect(volumeMountNames(mounts)).To(ContainElement("link-nats-deployment-nats-nats"))
+					Expect(volumeMountNames(mounts)).To(ContainElement("link-nats-deployment-type-name"))
 				}
 			})
 		})
@@ -176,12 +169,11 @@ var _ = Describe("Entangled Pods PodMutator", func() {
 				Expect(err).NotTo(HaveOccurred())
 
 				Expect(p.Spec.Volumes).To(HaveLen(2))
-				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats"))
-				Expect(volumeKeyToPaths(p.Spec.Volumes)).To(ContainElement("nats.nats"))
+				Expect(volumeNames(p.Spec.Volumes)).To(ContainElement("link-nats-deployment-nats-nats"))
 
 				for _, c := range p.Spec.Containers {
 					Expect(c.VolumeMounts).To(HaveLen(2))
-					Expect(volumeMountNames(c.VolumeMounts)).To(ContainElement("link-nats-deployment-nats"))
+					Expect(volumeMountNames(c.VolumeMounts)).To(ContainElement("link-nats-deployment-nats-nats"))
 				}
 			})
 		})

integration/quarks_link_from_bosh_test.go

@@ -7,14 +7,14 @@ import (
 	corev1 "k8s.io/api/core/v1"
 
 	bm "code.cloudfoundry.org/cf-operator/testing/boshmanifest"
+	"code.cloudfoundry.org/quarks-utils/pkg/names"
 	"code.cloudfoundry.org/quarks-utils/testing/machine"
 )
 
 var _ = Describe("BOSHLinks", func() {
 	const (
 		manifestRef    = "manifest"
 		deploymentName = "test"
-		secretName     = "link-test-nats"
 	)
 
 	var (
@@ -43,12 +43,18 @@ var _ = Describe("BOSHLinks", func() {
 		})
 
 		It("creates a secret for each link found in jobs", func() {
+			secretName := names.QuarksLinkSecretName(deploymentName, "nats", "nats")
+
 			By("waiting for secrets", func() {
 				err := env.WaitForSecret(env.Namespace, secretName)
 				Expect(err).NotTo(HaveOccurred())
 				secret, err := env.GetSecret(env.Namespace, secretName)
 				Expect(err).NotTo(HaveOccurred())
-				Expect(secret.Data).Should(HaveKeyWithValue("nats.nats", []byte("{\"nats\":{\"password\":\"changeme\",\"port\":4222,\"user\":\"admin\"}}")))
+				Expect(secret.Data).To(Equal(map[string][]byte{
+					"nats.password": []byte("changeme"),
+					"nats.port":     []byte("4222"),
+					"nats.user":     []byte("admin"),
+				}))
 			})
 		})
 	})
@@ -59,12 +65,18 @@ var _ = Describe("BOSHLinks", func() {
 		})
 
 		It("creates a secret for each link found in jobs", func() {
+			secretName := names.QuarksLinkSecretName(deploymentName, "nats", "nuts")
+
 			By("waiting for secrets", func() {
 				err := env.WaitForSecret(env.Namespace, secretName)
 				Expect(err).NotTo(HaveOccurred())
 				secret, err := env.GetSecret(env.Namespace, secretName)
 				Expect(err).NotTo(HaveOccurred())
-				Expect(secret.Data).Should(HaveKeyWithValue("nats.nuts", []byte("{\"nats\":{\"password\":\"changeme\",\"port\":4222,\"user\":\"admin\"}}")))
+				Expect(secret.Data).To(Equal(map[string][]byte{
+					"nats.password": []byte("changeme"),
+					"nats.port":     []byte("4222"),
+					"nats.user":     []byte("admin"),
+				}))
 			})
 		})
 	})

pkg/bosh/converter/job_factory.go

@@ -179,7 +179,7 @@ func (f *JobFactory) InstanceGroupManifestJob(manifest bdm.Manifest, linkInfos L
 		if ig.Instances != 0 {
 			// Additional secret for BOSH links per instance group
 			containerName := names.Sanitize(ig.Name)
-			linkOutputs[containerName] = names.EntanglementSecretName(manifest.Name, ig.Name)
+			linkOutputs[containerName] = names.QuarksLinkSecretName(manifest.Name)
 
 			// One container per instance group
 			containers = append(containers, ct.newUtilContainer(ig.Name, linkInfos.VolumeMounts()))
@@ -195,9 +195,11 @@ func (f *JobFactory) InstanceGroupManifestJob(manifest bdm.Manifest, linkInfos L
 	// add the BOSH link secret to the output list of each container
 	for container, secret := range linkOutputs {
 		qJob.Spec.Output.OutputMap[container]["provides.json"] = qjv1a1.SecretOptions{
-			Name: secret,
+			Name:              secret,
+			PersistenceMethod: qjv1a1.PersistUsingFanOut,
 		}
 	}
+
 	return qJob, nil
 }
 

pkg/bosh/converter/job_factory_test.go

@@ -90,29 +90,39 @@ var _ = Describe("JobFactory", func() {
 								Name:                   "foo-deployment.ig-resolved.redis-slave",
 								AdditionalSecretLabels: map[string]string{"quarks.cloudfoundry.org/secret-type": "ig-resolved"},
 								Versioned:              true,
+								PersistenceMethod:      "",
 							},
 							"bpm.json": qjv1a1.SecretOptions{
 								Name:                   "foo-deployment.bpm.redis-slave",
 								AdditionalSecretLabels: map[string]string{"quarks.cloudfoundry.org/secret-type": "bpm"},
 								Versioned:              true,
+								PersistenceMethod:      "",
 							},
 							"provides.json": qjv1a1.SecretOptions{
-								Name: "link-foo-deployment-redis-slave",
+								Name:                   "link-foo-deployment",
+								AdditionalSecretLabels: nil,
+								Versioned:              false,
+								PersistenceMethod:      "fan-out",
 							},
 						},
 						"diego-cell": qjv1a1.FilesToSecrets{
 							"ig.json": qjv1a1.SecretOptions{
 								Name:                   "foo-deployment.ig-resolved.diego-cell",
 								AdditionalSecretLabels: map[string]string{"quarks.cloudfoundry.org/secret-type": "ig-resolved"},
 								Versioned:              true,
+								PersistenceMethod:      "",
 							},
 							"bpm.json": qjv1a1.SecretOptions{
 								Name:                   "foo-deployment.bpm.diego-cell",
 								AdditionalSecretLabels: map[string]string{"quarks.cloudfoundry.org/secret-type": "bpm"},
 								Versioned:              true,
+								PersistenceMethod:      "",
 							},
 							"provides.json": qjv1a1.SecretOptions{
-								Name: "link-foo-deployment-diego-cell",
+								Name:                   "link-foo-deployment",
+								AdditionalSecretLabels: nil,
+								Versioned:              false,
+								PersistenceMethod:      "fan-out",
 							},
 						},
 					},