[Feature/Operator] Dataflow lifecycle management (#31)

* Fix tutorial version (#16)

* fix tutorial version

* fix blog post

Signed-off-by: erdrix <[email protected]>

* [Fix/Documentation] Chart helm CRDs and some documentations (#18)

* fix documentation

Signed-off-by: erdrix <[email protected]>

* clean tool

Signed-off-by: erdrix <[email protected]>

* add v1beta1 version

Signed-off-by: erdrix <[email protected]>

* remove crds from chart and add v1beta1

Signed-off-by: erdrix <[email protected]>

* update changelog

Signed-off-by: erdrix <[email protected]>

* clean

Signed-off-by: erdrix <[email protected]>

* Bump lodash from 4.17.15 to 4.17.19 in /website

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.15...4.17.19)

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: erdrix <[email protected]>

* Bump acorn from 6.4.0 to 6.4.1 in /website

Bumps [acorn](https://github.com/acornjs/acorn) from 6.4.0 to 6.4.1.
- [Release notes](https://github.com/acornjs/acorn/releases)
- [Commits](acornjs/acorn@6.4.0...6.4.1)

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: erdrix <[email protected]>

* tmp

Signed-off-by: erdrix <[email protected]>

* [Fix/Documentation] Pattern & versions (#24)

* fix documentation

Signed-off-by: erdrix <[email protected]>

* append changelog

* append changelog

Signed-off-by: erdrix <[email protected]>

* move v1 into specific folder

Signed-off-by: erdrix <[email protected]>

* add nifi registry client controller

Signed-off-by: erdrix <[email protected]>

* nifi dataflow skeleton

Signed-off-by: erdrix <[email protected]>

* [Feature/Chart] Quick fix for supporting watching multiple namespaces (#25)

* Update role_binding.yaml

* Update role.yaml

* Update values.yaml

* Add non clustered role as described in  #9

Signed-off-by: arttii <[email protected]>

* Add changelog

Signed-off-by: arttii <[email protected]>

* Add missing list to string conversion

Signed-off-by: arttii <[email protected]>
Signed-off-by: erdrix <[email protected]>

* refactor registry client reconcile

Signed-off-by: erdrix <[email protected]>

* clean config

Signed-off-by: erdrix <[email protected]>

* improve dataflow struct and reconcile skeleton

Signed-off-by: erdrix <[email protected]>

* refactor error management

Signed-off-by: erdrix <[email protected]>

* [Fix/Operator] PKI Finalizer namespace (#28)

* fix finalizer namespace

* append changelog

Signed-off-by: erdrix <[email protected]>

* append changelog

Signed-off-by: erdrix <[email protected]>

* prepare release 0.2.1 (#29)

Signed-off-by: erdrix <[email protected]>

* drop queues

Signed-off-by: erdrix <[email protected]>

* drop & drain, remove

Signed-off-by: erdrix <[email protected]>

* drop & drain, remove

Signed-off-by: erdrix <[email protected]>

* fix dataflow lifecycle

Signed-off-by: erdrix <[email protected]>

* reformat and fix

Signed-off-by: erdrix <[email protected]>

* automate crds generation

Signed-off-by: erdrix <[email protected]>

* fix ordering

Signed-off-by: erdrix <[email protected]>

* conf samples

Signed-off-by: erdrix <[email protected]>

* unit test for clients

Signed-off-by: erdrix <[email protected]>

* add documentation

Signed-off-by: erdrix <[email protected]>

* unit test for clients

Signed-off-by: erdrix <[email protected]>

* add documentation

Signed-off-by: erdrix <[email protected]>

* append changelog

Signed-off-by: erdrix <[email protected]>

* cleaning

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Arttii <[email protected]>

CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ### Added
 
+- [PR #31](https://github.com/Orange-OpenSource/nifikop/pull/31) - **[Operator]** Dataflow lifecycle management
+
 ### Changed
 
 ### Deprecated

Makefile

@@ -104,7 +104,11 @@ generate:
 	echo "Generate zzz-deepcopy objects"
 	operator-sdk version
 	operator-sdk generate k8s
+	mkdir -p deploy/crds/v1beta1 deploy/crds/v1
 	operator-sdk generate crds
+	mv deploy/crds/nifi* deploy/crds/v1
+	operator-sdk generate crds --crd-version v1beta1
+	mv deploy/crds/nifi* deploy/crds/v1beta1
 
 # Build nifikop executable file in local go env
 .PHONY: build

README.md

@@ -34,13 +34,13 @@ Some of the high-level capabilities and objectives of Apache NiFi include, and s
 - graceful NiFi cluster **scaling**
 - encrypted communication using SSL
 - the provisioning of secure NiFi clusters
+- Advanced Dataflow and user management via CRD
 
 Some of the roadmap features :
 
 - Monitoring via **Prometheus**
 - Automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins)
 - graceful NiFi cluster **scaling and rebalancing**
-- Advanced Dataflow and user management via CRD
 
 ## Motivation
 

config/samples/dataflow/dataflow-dev.yaml

@@ -0,0 +1,22 @@
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiDataflow
+metadata:
+  name: dataflow-lifecycle-dev
+spec:
+  parentProcessGroupID: "16cfd2ec-0174-1000-0000-00004b9b35cc"
+  bucketId: "01ced6cc-0378-4893-9403-f6c70d080d4f"
+  flowId: "8fefc874-0c93-4ce5-ba53-1672f0b59aaa"
+  flowVersion: 5
+  runOnce: true
+  skipInvalidControllerService: true
+  skipInvalidComponent: true
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  registryClientRef:
+    name: squidflow
+    namespace: nifikop
+  parameterContextRef:
+    name: dataflow-lifecycle-dev
+    namespace: nifikop
+  updateStrategy: drop

config/samples/dataflow/demo-config-dev.yaml

@@ -0,0 +1,26 @@
+# kubectl create secret generic secret-params --from-literal=secret1=yop --from-literal=secret2=yep
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiParameterContext
+metadata:
+  name: dataflow-lifecycle-dev
+spec:
+  description: "Contains dev configuration"
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  secretRefs:
+    - name: secret-params-dev
+      namespace: nifikop
+  parameters:
+    - name: value1
+      value: dev3
+      description: "First variable"
+    - name: value2
+      value: dev
+      description: "Second variable"
+    - name: value3
+      value: dev
+      description: "Third variable"
+    - name: value4
+      value: dev
+      description: "Fourth variable"

config/samples/dataflow/prod/dataflow-prod.yaml

@@ -0,0 +1,22 @@
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiDataflow
+metadata:
+  name: dataflow-lifecycle-prod
+spec:
+  parentProcessGroupID: "4dbcb875-0174-1000-217b-30a14fdd33ef"
+  bucketId: "01ced6cc-0378-4893-9403-f6c70d080d4f"
+  flowId: "8fefc874-0c93-4ce5-ba53-1672f0b59aaa"
+  flowVersion: 7
+  runOnce: false
+  skipInvalidControllerService: true
+  skipInvalidComponent: true
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  registryClientRef:
+    name: squidflow
+    namespace: nifikop
+  parameterContextRef:
+    name: dataflow-lifecycle-prod
+    namespace: nifikop
+  updateStrategy: drain

config/samples/dataflow/prod/demo-config-prod.yaml

@@ -0,0 +1,26 @@
+# kubectl create secret generic secret-params --from-literal=secret1=yop --from-literal=secret2=yep
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiParameterContext
+metadata:
+  name: dataflow-lifecycle-prod
+spec:
+  description: "Contains prod configuration"
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  secretRefs:
+    - name: secret-params-prod
+      namespace: nifikop
+  parameters:
+    - name: value1
+      value: prod1
+      description: "First variable"
+    - name: value2
+      value: prod2
+      description: "Second variable"
+    - name: value3
+      value: prod3
+      description: "Third variable"
+    - name: value4
+      value: prod4
+      description: "Fourth variable"

config/samples/dataflow/prod/secret-prod.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-params-prod
+type: Opaque
+data:
+  secret1: toto
+  secret2: titi
+  secret3: tutu

config/samples/dataflow/secret-dev.yaml

@@ -0,0 +1,10 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-params-dev
+type: Opaque
+data:
+  secret1: toto
+  secret2: titi
+  secret5: titi
+  secret4: tuot

config/samples/dataflow/squidflow-registry.yaml

@@ -0,0 +1,10 @@
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiRegistryClient
+metadata:
+  name: squidflow
+spec:
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  description: "Squidflow demo test"
+  uri: "http://nifi-registry-service:18080"

config/samples/nifi.orange.com_v1alpha1_nifidataflow.yaml

@@ -0,0 +1,33 @@
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiDataflow
+metadata:
+  name: dataflow-lifecycle
+spec:
+  # the UUID of the parent process group where you want to deploy your dataflow, if not set deploy at root level.
+  parentProcessGroupID: "16cfd2ec-0174-1000-0000-00004b9b35cc"
+  # the UUID of the Bucket containing the flow.
+  bucketId: "01ced6cc-0378-4893-9403-f6c70d080d4f"
+  # the UUID of the flow to run.
+  flowId: "9b2fb465-fb45-49e7-94fe-45b16b642ac9"
+  # the version of the flow to run, if not present or equals to -1, then the latest version of flow will be used.
+  flowVersion: 2
+  # if the flow will be ran once or continuously checked
+  runOnce: false
+  # whether the flow is considered as ran if some controller services are still invalid or not.
+  skipInvalidControllerService: true
+  #  whether the flow is considered as ran if some components are still invalid or not.
+  skipInvalidComponent: true
+  # contains the reference to the NifiCluster with the one the dataflow is linked.
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  # contains the reference to the NifiRegistry with the one the dataflow is linked.
+  registryClientRef:
+    name: squidflow
+    namespace: nifikop
+  # contains the reference to the ParameterContext with the one the dataflow is linked.
+  parameterContextRef:
+    name: dataflow-lifecycle
+    namespace: nifikop
+  # describes the way the operator will deal with data when a dataflow will be updated : drop or drain
+  updateStrategy: drain

config/samples/nifi.orange.com_v1alpha1_nifiparametercontext.yaml

@@ -0,0 +1,27 @@
+# kubectl create secret generic secret-params --from-literal=secret1=yop --from-literal=secret2=yep
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiParameterContext
+metadata:
+  name: dataflow-lifecycle
+spec:
+  # the Description of the Parameter Context.
+  description: "It is a test"
+  # contains the reference to the NifiCluster with the one the user is linked
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  # a list of secret containing sensitive parameters (the key will name of the parameter)
+  secretRefs:
+    - name: secret-params
+      namespace: nifikop
+  # a list of non-sensitive Parameters.
+  parameters:
+      # the name of the Parameter.
+    - name: test
+      # the value of the Parameter.
+      value: toto
+      # the description of the Parameter.
+      description: tutu
+    - name: test2
+      value: toto
+      description: toto

config/samples/nifi.orange.com_v1alpha1_nifiregistryclient.yaml

@@ -0,0 +1,13 @@
+apiVersion: nifi.orange.com/v1alpha1
+kind: NifiRegistryClient
+metadata:
+  name: squidflow
+spec:
+  # Contains the reference to the NifiCluster with the one the registry client is linked.
+  clusterRef:
+    name: nc
+    namespace: nifikop
+  # The Description of the Registry client.
+  description: "Squidflow demo"
+  # The URI of the NiFi registry that should be used for pulling the flow.
+  uri: "http://nifi-registry:18080"

config/samples/secret.yaml

@@ -0,0 +1,9 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: secret-params
+type: Opaque
+data:
+  value1: toto
+  value2: titi
+  value3: tutu