-
Notifications
You must be signed in to change notification settings - Fork 47
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
64 changed files
with
5,650 additions
and
17 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # nifikop 1.11.3 | ||
| # nifikop 1.11.4 | ||
| rbacEnable: true | ||
| namespaces: | ||
| - nifi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 34 additions & 0 deletions
34
site/website/versioned_docs/version-v1.11.4/1_concepts/1_start_here.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| id: 1_start_here | ||
| title: Start here | ||
| sidebar_label: Start here | ||
| --- | ||
|
|
||
| The Konpyūtāika NiFi operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache NiFi](https://nifi.apache.org/) clusters deployed to K8s. | ||
|
|
||
| ## Overview | ||
|
|
||
| Apache NiFi is an open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic. | ||
| Some of the high-level capabilities and objectives of Apache NiFi include, and some of the main features of the **NiFiKop** are: | ||
|
|
||
| - **Fine grained** node configuration support | ||
| - Graceful rolling upgrade | ||
| - graceful NiFi cluster **scaling** | ||
| - NiFi cluster **auto-scaling** | ||
| - Encrypted communication using SSL | ||
| - the provisioning of secure NiFi clusters | ||
| - Advanced Dataflow and user management via CRD | ||
|
|
||
| Some of the roadmap features: | ||
|
|
||
| - Automatic reaction and self healing based on alerts (plugin system, with meaningful default alert plugins) | ||
| - graceful NiFi cluster **rebalancing** | ||
|
|
||
| ## Motivation | ||
|
|
||
| There are already some approaches to operating NiFi on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our needs. | ||
|
|
||
| - [Helm chart](https://github.com/cetic/helm-nifi) | ||
| - [Cloudera Nifi Operator](https://blog.cloudera.com/cloudera-flow-management-goes-cloud-native-with-apache-nifi-on-red-hat-openshift-kubernetes-platform/) | ||
|
|
||
| Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. |
53 changes: 53 additions & 0 deletions
53
site/website/versioned_docs/version-v1.11.4/1_concepts/2_design_principles.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| --- | ||
| id: 2_design_principles | ||
| title: Design Principles | ||
| sidebar_label: Design Principles | ||
| --- | ||
|
|
||
| This operator is built following the logic implied by the [operator sdk framework](https://sdk.operatorframework.io/). | ||
| What we want to offer with NiFiKop is that we provide as much automation as possible to manage NiFi at scale and in the most automated way possible. | ||
|
|
||
| ## Separation of concerns | ||
|
|
||
| Kubernetes is designed for automation. Right out of the box, the Kubernetes core has a lot of automation features built in. You can use Kubernetes to automate the deployment and execution of workloads, and you can automate how Kubernetes does it. | ||
|
|
||
| The Kubernetes operator model concept allows you to extend cluster behavior without changing the Kubernetes code itself by binding controllers to one or more custom resources. Operators are clients of the Kubernetes API that act as controllers for a custom resource. | ||
|
|
||
| There are two things we can think of when we talk about operators in Kubernetes: | ||
|
|
||
| - Automate the deployment of my entire stack. | ||
| - Automate the actions required by the deployment. | ||
|
|
||
| For NiFiKop, we focus primarily on NiFi for the stack concept, what does that mean? | ||
|
|
||
| - We do not manage other components that can be integrated with NiFi Cluster like Prometheus, Zookeeper, NiFi registry etc. | ||
| - We want to provide as many tools as possible to automate the work on NiFi (cluster deployment, data flow and user management, etc.). | ||
|
|
||
| We consider that for NiFiKop to be a low-level operator, focused on NiFi and only NiFi, and if we want to manage a complex stack with e.g. Zookeeper, NiFi Registry, Prometheus etc. we need something else working at a higher level, like Helm charts or another operator controlling NiFiKop and other resources. | ||
|
|
||
| ## One controller per resource | ||
|
|
||
| The operator should reflect as much as possible the behavior of the solution we want to automate. If we take our case with NiFi, what we can say is that: | ||
|
|
||
| - You can have one or more NiFi clusters | ||
| - On your cluster you can define a NiFi registry client, but it is not mandatory. | ||
| - You can also define users and groups and deploy a DataFlow if you want. | ||
|
|
||
| This means that your cluster is not defined by what is deployed on it, and what you deploy on it does not depend on it. | ||
| To be more explicit, just because I deploy a NiFi cluster doesn't mean the DataFlow deployed on it will stay there, we can move the DataFlow from one cluster to another. | ||
|
|
||
| To manage this, we need to create different kinds of resources ([NifiCluster], [NifiDataflow], [NifiParameterContext], [NifiUser], [NifiUserGroup], [NifiRegistryClient], [NifiNodeGroupAutoscaler], [NifiConnection]) with one controller per resource that will manage its own resource. | ||
| In this way, we can say: | ||
|
|
||
| - I deploy a NiFiCluster | ||
| - I define a NiFiDataflow that will be deployed on this cluster, and if I want to change cluster, I just have to change the `ClusterRef` to change cluster | ||
|
|
||
|
|
||
| [NifiCluster]: ../5_references/1_nifi_cluster | ||
| [NifiDataflow]: ../5_references/5_nifi_dataflow | ||
| [NifiParameterContext]: ../5_references/4_nifi_parameter_context | ||
| [NifiUser]: ../5_references/2_nifi_user | ||
| [NifiUserGroup]: ../5_references/6_nifi_usergroup | ||
| [NifiRegistryClient]: ../5_references/3_nifi_registry_client | ||
| [NifiNodeGroupAutoscaler]: ../5_references/7_nifi_nodegroup_autoscaler | ||
| [NifiConnection]: ../5_references/8_nifi_connection |
67 changes: 67 additions & 0 deletions
67
site/website/versioned_docs/version-v1.11.4/1_concepts/3_features.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| --- | ||
| id: 3_features | ||
| title: Features | ||
| sidebar_label: Features | ||
| --- | ||
|
|
||
| To highligt some of the features we needed and were not possible with the operators available, please keep reading | ||
|
|
||
| ## Fine Grained Node Config Support | ||
|
|
||
| We needed to be able to react to events in a fine-grained way for each Node - and not in the limited way StatefulSet does (which, for example, removes the most recently created Nodes). Some of the available solutions try to overcome these deficits by placing scripts inside the container to generate configs at runtime (a good example is our [Cassandra Operator](https://github.com/Orange-OpenSource/casskop)), whereas the Orange NiFi operator's configurations are deterministically placed in specific Configmaps. | ||
|
|
||
| ## Graceful NiFi Cluster Scaling | ||
|
|
||
| Apache NiFi is a good candidate to create an operator, because everything is made to orchestrate it through REST Api calls. With this comes automation of actions such as scaling, following all required steps: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#decommission-nodes. | ||
|
|
||
| ## Graceful Rolling Upgrade | ||
|
|
||
| Operator supports graceful rolling upgrade. It means the operator will check if the cluster is healthy. | ||
|
|
||
| ## Dynamic Configuration Support | ||
|
|
||
| NiFi operates with two type of configs: | ||
|
|
||
| - Read-only | ||
| - PerNode | ||
|
|
||
| Read only config requires node restart to update all the others may be updated dynamically. | ||
| Operator CRD distinguishes these fields, and proceed with the right action. It can be a rolling upgrade, or | ||
| a dynamic reconfiguration. | ||
|
|
||
| ## Dataflow lifecycle management via CRD | ||
|
|
||
| In a cloud native approach, we are looking for important management features, which we have applied to NiFi Dataflow: | ||
|
|
||
| - **Automated deployment:** Based on the NiFi registry, you can describe your `NiFiDataflow` resource that will be deployed and run on the targeted NiFi cluster. | ||
| - **Portability:** On kubernetes everything is a yaml file, so with NiFiKop we give you the ability to describe your clusters but also the `registry clients`, `parameter contexts` and `dataflows` of your NiFi application, so that you can redeploy the same thing in a different namespace or cluster. | ||
| - **State management:** With NiFiKop resources, you can describe what you want, and the operator deals with the NiFi Rest API to make sure the resource stays in sync (even if someone manually makes changes directly on NiFi cluster). | ||
| - **Configurations:** Based on the `Parameter Contexts`, NiFiKop allows you to associate to your `Dataflow` (= your applications) with a different configuration depending on the environment ! | ||
|
|
||
| ## Users and access policies management | ||
|
|
||
| Without the management of users and access policies associated, it was not possible to have a fully automated NiFi cluster setup due to: | ||
|
|
||
| - **Node scaling:** when a new node joins the cluster it needs to have some roles like `proxy user request`, `view data` etc., by managing users and access policies we can easily create a user for this node with the right accesses. | ||
| - **Operator admin rights:** For the operator to manage efficiently the cluster it needs a lot of rights as `deploying process groups`, `empty the queues` etc., these rights are not available by default when you set a user as [InitialAdmin](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity). Once again by giving the ability to define users and access policies we go through this. | ||
| - **User's access:** as seen just below we need to define the operator as `InitialAdmin`, in this situation there is no more users that can access to the web UI to manually give access to other users. That's why we extend the `InitialAdmin` concept into the operator, giving the ability to define a list of users as admins. | ||
|
|
||
| In addition to these requirements to have a fully automated and managed cluster, we introduced some useful features: | ||
|
|
||
| - **User management:** using `NifiUser` resource, you are able to create (or bind an existing) user in NiFi cluster and apply some access policies that will be managed and continuously synced by the operator. | ||
| - **Group management:** using `NifiUserGroup` resource, you can create groups in NiFi cluster and apply access policies and a list of `NifiUser` that will be managed and continuously synced by the operator. | ||
| - **Default group:** As the definition of `NifiUser` and `NifiUserGroup` resources could be heavy for some simple use cases, we also decided to define two default groups that you can feed with a list of users that will be created and managed by the operator (no kubernetes resources to create): | ||
| - **Admins:** a group giving access to everything on the NiFi Cluster, | ||
| - **Readers:** a group giving access as viewer on the NiFi Cluster. | ||
|
|
||
| By introducing this feature we are giving you the ability to fully automate your deployment, from the NiFi Cluster to your managed NiFi Dataflow. | ||
|
|
||
| ## Automatic horizontal NiFi cluster scaling via CRD | ||
|
|
||
| NiFiKop supports automatically horizontally scaling `NifiCluster` node groups with a `NifiNodeGroupAutoscaler` custom resource. | ||
|
|
||
| - **Kubernetes native:** The `NifiNodeGroupAutoscaler` controller implements the [Kubernetes scale subresource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource) and creates a Kubernetes [`HorizontalPodAutoscaler`](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) to automatically scale the pods that NiFiKop creates for `NifiCluster` deployments. | ||
| - **Metrics-driven autoscaling:** The `HorizontalPodAutoscaler` can be driven by pod usage metrics (e.g. RAM, CPU) or through NiFi application metrics scraped by Prometheus. | ||
| - **Flexible NifiCluster node group autoscaling:** The `NifiNodeGroupAutoscaler` scales specific node groups in your `NifiCluster` and you may have as many autoscalers as you like per `NifiCluster` deployment. For example, a `NifiNodeGroupAutoscaler` may manage high-memory or high-cpu sets of nodes for volume burst scenarios or it may manage every node in your cluster. | ||
|
|
||
| Through this set of features, you may elect to have NiFiKop configure automatic horizontal autoscaling for any subset of nodes in your `NifiCluster` deployment. |
Oops, something went wrong.