This project contains the manifests that are used to deploy highly available Elasticsearch cluster along with the monitoring tools that are used to follow some metrics to the Kubernetes.
Kubernetes version is v1.14.1 and Docker version is 18.9.2 for all of the worker nodes.
Helm is used to deploy Kubernetes resources. Helm provides important facilities. These are;
- Installing a detailed application to the Kubernetes with one command
- Upgrading an application that is already deployed to the Kubernetes
- Rolling back an application deployment and return to a previous release when something goes wrong
Prometheus, Node Exporter, Kube State Metrics, Grafana stack is used to monitor resource utilisation of Kubernetes nodes and pods that are running in Kubernetes.
Fluentd and Kibana are used to aggregate and visualize the pod logs across the Kubernetes cluster. They are connecting to the highly available Elasticsearch cluster.
- kubectl cli v1.14.1
- Access to a Kubernetes cluster.
- Valid config file under the $HOME/.kube/ directory named as config.
- Docker as container runtime in your Kubernetes nodes.
Validate your kubectl installation with the following command;
kubectl get nodes -owide
NAME STATUS VERSION OS-IMAGE CONTAINER-RUNTIME
pool-goa4adyjv-qv05 Ready v1.14.1 Debian GNU/Linux 9 (stretch) docker://18.9.2
.....
.....Change your directory to root folder of this project. And then execute the following command to install Helm command line tool to your local environment and deploy Tiller to the Kubernetes;
bash install-helm.shWait until the Tiller deployment is ready;
kubectl get deploy -n kube-system | grep -i tiller
NAME READY UP-TO-DATE AVAILABLE AGE
tiller-deploy 1/1 1 1 10hExecute the following command to deploy highly available Elasticsearch cluster and monitoring tools to the Kubernetes:
# Set a map in the terminal that contains the chart directories as keys and namespaces as values
charts=(
"elasticsearch:ha-elasticsearch-cluster"
"fluentd:kube-logging"
"kibana:kube-logging"
"prometheus:kube-monitoring"
"node-exporter:kube-monitoring"
"kube-state-metrics:kube-monitoring"
"grafana:kube-monitoring"
)
# Then execute the script to deploy the charts
bash deploy-all-charts.sh "${charts[@]}"It will deploy the Helm charts for the following applications/tools;
- Elasticsearch
- Prometheus
- Kube State Metrics
- Node Exporter
- Grafana
- Fluentd
- Kibana
and creates the following namespaces;
- ha-elasticsearch-cluster
- kube-logging
- kube-monitoring
Verify the installation by executing the following commands:
helm ls
NAME REVISION STATUS
elasticsearch 1 DEPLOYED ....
fluentd 1 DEPLOYED ....
grafana 1 DEPLOYED ....
kibana 1 DEPLOYED ....
kube-state-metrics 1 DEPLOYED ....
node-exporter 1 DEPLOYED ....
prometheus 1 DEPLOYED ....
kubectl get pods -n ha-elasticsearch-cluster
NAME READY STATUS
ha-elasticsearch-0 1/1 Running
ha-elasticsearch-1 1/1 Running
ha-elasticsearch-2 1/1 Running
kubectl get pods -n kube-logging
NAME READY STATUS
fluentd-hlf6k 1/1 Running
fluentd-kpcvd 1/1 Running
fluentd-l5rnw 1/1 Running
kibana-7bc7984bf5-pmfql 1/1 Running
kubectl get pods -n kube-monitoring
NAME READY STATUS
grafana-8675988864-rbhgf 1/1 Running
kube-state-metrics-7986d88c55-jjfzn 2/2 Running
node-exporter-ct4ft 1/1 Running
node-exporter-gpmgt 1/1 Running
node-exporter-wlhhw 1/1 Running
prometheus-7b476bf58c-nvnwq 1/1 RunningThe Elasticsearch cluster can be accessed by using kubectl port-forward. To forward Elasticsearch service port to your local environment, execute the following command;
kubectl port-forward ha-elasticsearch-0 9200:9200 -n ha-elasticsearch-clusterThen test it with the following command;
curl -XGET http://localhost:9200/_cluster/stats?prettyThere is a values.yaml file in each Helm chart directory. You can edit this file in case you want to deploy a chart with a different config.
Since Elasticsearch is a stateful service, its chart will be deployed as StatefulSet. Each pod of the StatefulSet will be scheduled on different worker nodes. Since I have 3 worker nodes, I have set the replica count to 3 in the values.yaml. This count can be increased as we join more worker nodes to the Kubernetes cluster. Here are the environment variables used below;
-
ES_JAVA_OPTS will be set to "-Xms4g -Xmx4g" since I have 8 GB of memory in each Kubernetes worker nodes.
-
discovery.seed_hosts will be set to node names of the Elasticsearch cluster to provide a list of other nodes in the cluster that are master-eligible and contactable. Kubernetes service dns names are used to provide hostnames. Its value is automatically generated from the values.yaml and helpers.tpl files. In my case the value is
ha-elasticsearch-0.ha-elasticsearch,ha-elasticsearch-1.ha-elasticsearch,ha-elasticsearch-2.ha-elasticsearch. Here it will use the default 9300 port to connect to Elasticsearch cluster nodes since I am providing only the hostnames. -
cluster.initial_master_nodes will be set to the node names of the Elasticsearch cluster. All of the Elasticsearch nodes will start as master-eligible. In my case the value is
ha-elasticsearch-0.ha-elasticsearch,ha-elasticsearch-1.ha-elasticsearch,ha-elasticsearch-2.ha-elasticsearch
While increasing the replica count of the StatefulSet, we need to consider the "split-brain" issue. Number of master-eligible nodes should be an odd number. Also discovery.zen.minimum_master_nodes environment variable should set to N/2 + 1, and rounding the result down to the closest integer (where N is the number of master eligible nodes). With my current configuration discovery.zen.minimum_master_nodes is set to 2. Each Elasticsearch cluster instance is master-eligible since they will start running with the same role.
Normally, more than one Statefulset pods may locate in the same Kubernetes host. If more than one instance of the Elasticsearch cluster will schedule to the same Kubernetes host, and this host will become unavailable, the end users may experince an outage until the pod rescheduled to another host or host become available. To prevent such an outage, we can schedule each pod to a different Kubernetes host. In order to do this, I have added the following part to the Elasticsearch Statefulset manifest;
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- ha-elasticsearch
topologyKey: "kubernetes.io/hostname"
We can check the node names for each pod are different;
for i in 0 1 2;
do
kubectl get pod ha-elasticsearch-$i -n ha-elasticsearch-cluster --template {{.spec.nodeName}}; echo "";
done
# result
pool-goa4adyjv-qv0k
pool-goa4adyjv-qv05
pool-goa4adyjv-qv0hAlso I have created PodDisruptionBudget to indicate that at most one pod from the Elasticsearch StatefulSet can be unavailable at any time. You can change this value in the values.yaml
When we want to deploy a new version of our Elasticsearch chart or when we want to change the configs of it, we can use helm upgrade command. Also with the help of Kubernetes liveness and readiness probes, we can update our Elasticsearch cluster with no downtime.
helm upgrade -f values.yaml . --namespace ha-elasticsearch-cluster
helm ls | grep elasticsearch
NAME REVISION STATUS
elasticsearch 2 DEPLOYEDWe can see the revision count is incremented by one.
When something goes wrong with a deployment update, we may want to perform a rollback operation to return the last stable version of our deployment. With the help of Helm, rollback operation can be achieved easily by using helm rollback. When we perform an install, upgrade or rollback operation, Helm increments the revision number. Later on, we can use this revision number to rollback our deployment.
helm rollback elasticsearch 1
# And see the rollback effect
kubectl get pods -n ha-elasticsearch-cluster -wSince Elastichsearch is a stateful application, we need to monitor its disk usage. By monitoring the disk usage, we can take action to increase the disk size.
Also by monitoring the CPU and Memory consumption by the Kubernetes worker nodes, we can decide to expand the cluster by adding new worker nodes.
Prometheus and Grafana mainly used to monitor the cluster. Grafana chart will be deployed with preconfigured data source and dashboards. Currently you can view two different dashboards in Grafana;
- Kubernetes Cluster Monitoring
- Displays live memory/cpu/disk usages of each worker node.
- Kubernetes Namespace Monitoring
- Displays live memory/cpu/disk usages of each pod in the Kubernetes cluster by namespace.
Fluentd and Kibana mainly used to aggregate and visualize the pod logs across the Kubernetes cluster. They are connecting to the highly available Elasticsearch cluster. You can filter the logs by pods in this dashboard.
# Set an array in the terminal that contains chart directories
charts=(
"elasticsearch"
"fluentd"
"kibana"
"prometheus"
"node-exporter"
"kube-state-metrics"
"grafana"
)
# Then execute the script to delete the charts
bash delete-all-charts.sh "${charts[@]}"kubectl delete persistentvolumeclaims \
--namespace ha-elasticsearch-cluster \
--selector app=elasticsearchfor namespace in "ha-elasticsearch-cluster" "kube-logging" "kube-monitoring"; do
kubectl delete ns $namespace;
done-
Elasticsearch, Setting the Heap Size https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
-
Elasticsearch, Node Discovery Settings https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#unicast.hosts
-
Elasticsearch, Virtual Memory Maximum MMap Count https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html#vm-max-map-count
-
Elasticsearch, File Descriptors https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html#file-descriptors
-
Kubernetes, Tolerating Node Failure https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/#tolerating-node-failure