-
Notifications
You must be signed in to change notification settings - Fork 47
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Release] Bump version 0.14.0 (#158)
* Release bump version to 0.14.0 * Release bump version to 0.14.0
- Loading branch information
Showing
51 changed files
with
3,537 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # nifikop 0.13.0 | ||
| # nifikop 0.14.0 | ||
| rbacEnable: true | ||
| namespaces: | ||
| - nifi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 34 additions & 0 deletions
34
site/website/versioned_docs/version-v0.14.0/1_concepts/1_introduction.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| id: 1_introduction | ||
| title: Introduction | ||
| sidebar_label: Introduction | ||
| --- | ||
|
|
||
| The Konpyūtāika NiFi operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache NiFi](https://nifi.apache.org/) clusters deployed to K8s. | ||
|
|
||
| ## Overview | ||
|
|
||
| Apache NiFi is an open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic. | ||
| Some of the high-level capabilities and objectives of Apache NiFi include, and some of the main features of the **NiFiKop** are: | ||
|
|
||
| - **Fine grained** node configuration support | ||
| - Graceful rolling upgrade | ||
| - graceful NiFi cluster **scaling** | ||
| - Encrypted communication using SSL | ||
| - the provisioning of secure NiFi clusters | ||
| - Advanced Dataflow and user management via CRD | ||
|
|
||
| Some of the roadmap features : | ||
|
|
||
| - Monitoring via **Prometheus** | ||
| - Automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins) | ||
| - graceful NiFi cluster **scaling and rebalancing** | ||
|
|
||
| ## Motivation | ||
|
|
||
| There are already some approaches to operating NiFi on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our needs. | ||
|
|
||
| - [Helm chart](https://github.com/cetic/helm-nifi) | ||
| - [Cloudera Nifi Operator](https://blog.cloudera.com/cloudera-flow-management-goes-cloud-native-with-apache-nifi-on-red-hat-openshift-kubernetes-platform/) | ||
|
|
||
| Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. |
62 changes: 62 additions & 0 deletions
62
site/website/versioned_docs/version-v0.14.0/1_concepts/2_design_principes.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| id: 2_design_principes | ||
| title: Design Principes | ||
| sidebar_label: Design Principes | ||
| --- | ||
|
|
||
| ## Pod level management | ||
|
|
||
| NiFi is a stateful application. The first piece of the puzzle is the Node, which is a simple server capable of createing/forming a cluster with other Nodes. Every Node has his own **unique** configuration which differs slightly from all others. | ||
|
|
||
| All NiFi on Kubernetes setup use [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) to create a NiFi Cluster. Just to quickly recap from the K8s docs: | ||
|
|
||
| >StatefulSet manages the deployment and scaling of a set of Pods, and provide guarantees about their ordering and uniqueness. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains sticky identities for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that is maintained across any rescheduling. | ||
| How does this looks from the perspective of Apache NiFi ? | ||
|
|
||
| With StatefulSet we get : | ||
| - unique Node IDs generated during Pod startup | ||
| - networking between Nodes with headless services | ||
| - unique Persistent Volumes for Nodes | ||
|
|
||
| Using StatefulSet we **lose** the ability to : | ||
|
|
||
| - modify the configuration of unique Nodes | ||
| - remove a specific Node from a cluster (StatefulSet always removes the most recently created Node) | ||
| - use multiple, different Persistent Volumes for each Node | ||
|
|
||
| The NiFi Operator uses `simple` Pods, ConfigMaps, and PersistentVolumeClaims, instead of StatefulSet (based on the design used by [Banzai Cloud Kafka Operator](https://github.com/banzaicloud/kafka-operator)). | ||
| Using these resources allows us to build an Operator which is better suited to NiFi. | ||
|
|
||
| With the NiFi operator we can: | ||
|
|
||
| - modify the configuration of unique Nodes | ||
| - remove specific Nodes from clusters | ||
| - use multiple Persistent Volumes for each Node | ||
|
|
||
| ## Dataflow Lifecycle management | ||
|
|
||
| The [Dataflow Lifecycle management feature](./3_features.md#dataflow-lifecycle-management-via-crd) introduces 3 new CRDs : | ||
|
|
||
| - **NiFiRegistryClient :** Allowing you to declare a [NiFi registry client](https://nifi.apache.org/docs/nifi-registry-docs/html/getting-started.html#connect-nifi-to-the-registry). | ||
| - **NiFiParameterContext :** Allowing you to create parameter context, with two kinds of parameters, a simple `map[string]string` for non-sensitive parameters and a `list of secrets` which contains sensitive parameters. | ||
| - **NiFiDataflow :** Allowing you to declare a Dataflow based on a `NiFiRegistryClient` and optionally a `ParameterContext`, which will be deployed and managed by the operator on the `targeted NiFi cluster`. | ||
|
|
||
| The following diagram shows the interactions between all the components : | ||
|
|
||
|  | ||
|
|
||
| With each CRD comes a new controller, with a reconcile loop : | ||
|
|
||
| - **NiFiRegistryClient's controller :** | ||
|
|
||
|  | ||
|
|
||
| - **NiFiParameterContext's controller :** | ||
|
|
||
|  | ||
|
|
||
| - **NiFiDataflow's controller :** | ||
|
|
||
|  | ||
|
|
Oops, something went wrong.