diff --git a/README.md b/README.md index 47d7ed7..90bad06 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,72 @@ -# Monitoring Kubernetes clusters on AWS using Prometheus +# Monitoring Kubernetes clusters on AWS using Prometheus Operator by CoreOS ![alt](https://www.camil.org/content/images/2017/cluster.png) +**Note:** the work on this repository is now based on CoreOS's [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus) and it will be the default option for Kubernetes 1.7.X and up. For 1.5.X and 1.6.X you can deploy a simpler solution, located in `./basic` directory. +The purpose of this project is to provide a simple and interactive method to deploy and configure Prometheus on Kubernetes, especially for the users that are not using Helm. + ## Features -* Prometheus v2.X.X -* InCluster deployment using a `StatefulSet` for persistent storage -* auto-discovery for services and pods annotated with `prometheus.io/scrape: 'true'` -* automatic configuration for RBAC +* Prometheus Operator with support for Prometheus v2.X.X +* highly available Prometheus and Alertmaneger +* InCluster deployment using `StatefulSets` for persistent storage +* auto-discovery for services and pods +* automatic RBAC configuration * preconfigured alerts * preconfigured Grafana dashboards -* easy to setup; usually less than a minute to deploy a basic monitoring solution for Kubernetes -* support for Kubernetes v1.6.0 and up - - -If you prefer a much advanced monitoring solution based on [Prometheus Operator](https://github.com/coreos/prometheus-operator) please check the `./operator` directory. +* easy to setup; usually less than a minute to deploy a complete monitoring solution for Kubernetes +* support for Kubernetes v1.7.x and up ## One minute deployment -[![asciicast](https://asciinema.org/a/QdIFKxowJ9XOSpS9QYuGI23J5.png)](https://asciinema.org/a/QdIFKxowJ9XOSpS9QYuGI23J5) +[![asciicast](https://asciinema.org/a/139033.png)](https://asciinema.org/a/139033) ## Prerequisites * Kubernetes cluster and `kubectl` configured -* Security Groups configured to allow port 9100/TCP for `prometheus node-exporter` and 10250/TCP for k8s nodes metrics. +* Security Groups configured to allow the fallowing ports: + * 9100/TCP - node-exporter + * 10250/TCP - kubernetes nodes metrics, + * 10251/TCP - kube-scheduler + * 10252/TCP - kube-controller-manager + * 10054/TCP and 10055/TCP - kube-dns #### Optional * SMTP Account for email alerts -* Token for alerts on Slack -* A IAM Role with EC2 ReadOnly access for EC2 instances monitoring. Only required for monitoring AWS nodes that are not part of the kubernetes cluster +* Token for Slack alerts +## Pre-Deployment +Clone the repository and checkout the latest release: `curl -L https://git.io/getPrometheusKubernetes | sh -` -## Pre-Deployment -Clone repository +## Custom settings + +All the components versions can be configured using the interactive deployment script. Same for the SMTP account or the Slack token. + +Some other settings that can be changed before deployment: + * **Prometheus replicas:** default **2** ==> `manifests/prometheus/prometheus-k8s.yaml` + * **persistent volume size:** default **40Gi** ==> `manifests/prometheus/prometheus-k8s.yaml` + * **allocated memory for Prometheus pods:** default **2Gi** ==> `manifests/prometheus/prometheus-k8s.yaml` + * **Alertmanager replicas:** default **3** ==> `manifests/alertmanager/alertmanager.yaml` + * **Alertmanager configuration:** ==> `assets/alertmanager/alertmanager.yaml` + * **custom Grafana dashboards:** add yours in `assets/grafana/` with names ending in `-dashboard.json` + * **custom alert rules:** ==> `assets/prometheus/rules/` - git clone github.com/camilb/prometheus-kubernetes && cd prometehus-kubernetes +**Note:** please commit your changes before deployment if you wish to keep them. The `deploy` script will remove the changes on most of the files. -Make any desired configuration changes in `configmaps` according to your setup. -* ./k8s/prometheus/prometheus.cm.yaml -* ./k8s/prometheus/alertmanager.cm.yaml +## Deploy + ./deploy -## Deploy Prometheus, Alertmaneger, Node Exporter, Grafana and Kube State Metrics +Now you can access the dashboards locally using `kubectl port-forward`command, or expose the services using a ingress or a LoadBalancer. Please check the `./tools` directory to quickly configure a ingress or proxy the services to localhost. - ./init.sh +To remove everything, just execute the `./teardown` script. -* The init script will ask some basic questions and attempt to auto-discover information about your system. +## Updating configurations -Now you can access the dashboards locally using `kubectl port-forward`command, creating a ingress or a LoadBalancer. Please check the `./tools` directory to quickly configure a ingress or proxy the services to localhost. + * **update alert rules:** add or change the rules in `assets/prometheus/rules/` and execute `scripts/generate-rules-configmap.sh`. Then apply the changes using `kubectl apply -f manifests/prometheus/prometheus-k8s-rules.yaml -n monitoring` + * **update grafana dashboards:** add or change the existing dashboards in `assets/grafana/` and execute `scripts/generate-dashboards-configmap.sh`. Then apply the changes using `kubectl apply -f manifests/grafana/grafana-dashboards.cm.yaml`. -To remove everything, just execute the `./remove.sh` script. +**Note:** all the Grafana dashboards should have names ending in `-dashboard.json`. diff --git a/operator/assets/alertmanager/alertmanager.yaml b/assets/alertmanager/alertmanager.yaml similarity index 100% rename from operator/assets/alertmanager/alertmanager.yaml rename to assets/alertmanager/alertmanager.yaml diff --git a/operator/assets/grafana/deployment-dashboard.json b/assets/grafana/deployment-dashboard.json similarity index 100% rename from operator/assets/grafana/deployment-dashboard.json rename to assets/grafana/deployment-dashboard.json diff --git a/operator/assets/grafana/kubernetes-capacity-planing-dashboard.json b/assets/grafana/kubernetes-capacity-planing-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-capacity-planing-dashboard.json rename to assets/grafana/kubernetes-capacity-planing-dashboard.json diff --git a/operator/assets/grafana/kubernetes-cluster-health-dashboard.json b/assets/grafana/kubernetes-cluster-health-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-cluster-health-dashboard.json rename to assets/grafana/kubernetes-cluster-health-dashboard.json diff --git a/operator/assets/grafana/kubernetes-cluster-overview-dashboard.json b/assets/grafana/kubernetes-cluster-overview-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-cluster-overview-dashboard.json rename to assets/grafana/kubernetes-cluster-overview-dashboard.json diff --git a/operator/assets/grafana/kubernetes-cluster-status-dashboard.json b/assets/grafana/kubernetes-cluster-status-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-cluster-status-dashboard.json rename to assets/grafana/kubernetes-cluster-status-dashboard.json diff --git a/operator/assets/grafana/kubernetes-control-plane-status-dashboard.json b/assets/grafana/kubernetes-control-plane-status-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-control-plane-status-dashboard.json rename to assets/grafana/kubernetes-control-plane-status-dashboard.json diff --git a/operator/assets/grafana/kubernetes-resource-requests-dashboard.json b/assets/grafana/kubernetes-resource-requests-dashboard.json similarity index 100% rename from operator/assets/grafana/kubernetes-resource-requests-dashboard.json rename to assets/grafana/kubernetes-resource-requests-dashboard.json diff --git a/operator/assets/grafana/nodes-dashboard.json b/assets/grafana/nodes-dashboard.json similarity index 100% rename from operator/assets/grafana/nodes-dashboard.json rename to assets/grafana/nodes-dashboard.json diff --git a/operator/assets/grafana/pods-dashboard.json b/assets/grafana/pods-dashboard.json similarity index 100% rename from operator/assets/grafana/pods-dashboard.json rename to assets/grafana/pods-dashboard.json diff --git a/operator/assets/grafana/prometheus-datasource.json b/assets/grafana/prometheus-datasource.json similarity index 100% rename from operator/assets/grafana/prometheus-datasource.json rename to assets/grafana/prometheus-datasource.json diff --git a/operator/assets/prometheus/rules/alertmanager.rules b/assets/prometheus/rules/alertmanager.rules similarity index 100% rename from operator/assets/prometheus/rules/alertmanager.rules rename to assets/prometheus/rules/alertmanager.rules diff --git a/operator/assets/prometheus/rules/custom.rules b/assets/prometheus/rules/custom.rules similarity index 100% rename from operator/assets/prometheus/rules/custom.rules rename to assets/prometheus/rules/custom.rules diff --git a/operator/assets/prometheus/rules/etcd3.rules b/assets/prometheus/rules/etcd3.rules similarity index 100% rename from operator/assets/prometheus/rules/etcd3.rules rename to assets/prometheus/rules/etcd3.rules diff --git a/operator/assets/prometheus/rules/general.rules b/assets/prometheus/rules/general.rules similarity index 100% rename from operator/assets/prometheus/rules/general.rules rename to assets/prometheus/rules/general.rules diff --git a/operator/assets/prometheus/rules/kube-apiserver.rules b/assets/prometheus/rules/kube-apiserver.rules similarity index 100% rename from operator/assets/prometheus/rules/kube-apiserver.rules rename to assets/prometheus/rules/kube-apiserver.rules diff --git a/operator/assets/prometheus/rules/kube-controller-manager.rules b/assets/prometheus/rules/kube-controller-manager.rules similarity index 100% rename from operator/assets/prometheus/rules/kube-controller-manager.rules rename to assets/prometheus/rules/kube-controller-manager.rules diff --git a/operator/assets/prometheus/rules/kube-scheduler.rules b/assets/prometheus/rules/kube-scheduler.rules similarity index 100% rename from operator/assets/prometheus/rules/kube-scheduler.rules rename to assets/prometheus/rules/kube-scheduler.rules diff --git a/operator/assets/prometheus/rules/kubelet.rules b/assets/prometheus/rules/kubelet.rules similarity index 100% rename from operator/assets/prometheus/rules/kubelet.rules rename to assets/prometheus/rules/kubelet.rules diff --git a/operator/assets/prometheus/rules/kubernetes.rules b/assets/prometheus/rules/kubernetes.rules similarity index 100% rename from operator/assets/prometheus/rules/kubernetes.rules rename to assets/prometheus/rules/kubernetes.rules diff --git a/operator/assets/prometheus/rules/node.rules b/assets/prometheus/rules/node.rules similarity index 100% rename from operator/assets/prometheus/rules/node.rules rename to assets/prometheus/rules/node.rules diff --git a/operator/assets/prometheus/rules/prometheus.rules b/assets/prometheus/rules/prometheus.rules similarity index 100% rename from operator/assets/prometheus/rules/prometheus.rules rename to assets/prometheus/rules/prometheus.rules diff --git a/basic/README.md b/basic/README.md new file mode 100644 index 0000000..dfe3db3 --- /dev/null +++ b/basic/README.md @@ -0,0 +1,52 @@ +# Monitoring Kubernetes clusters on AWS using Prometheus + + +![alt](https://www.camil.org/content/images/2017/cluster.png) + +## Features +* Prometheus v2.X.X +* InCluster deployment using a `StatefulSet` for persistent storage +* auto-discovery for services and pods annotated with `prometheus.io/scrape: 'true'` +* automatic configuration for RBAC +* preconfigured alerts +* preconfigured Grafana dashboards +* easy to setup; usually less than a minute to deploy a basic monitoring solution for Kubernetes +* support for Kubernetes v1.6.0 and up + +## One minute deployment +[![asciicast](https://asciinema.org/a/QdIFKxowJ9XOSpS9QYuGI23J5.png)](https://asciinema.org/a/QdIFKxowJ9XOSpS9QYuGI23J5) + + +## Prerequisites + +* Kubernetes cluster and `kubectl` configured +* Security Groups configured to allow port 9100/TCP for `prometheus node-exporter` and 10250/TCP for k8s nodes metrics. + +#### Optional +* SMTP Account for email alerts +* Token for alerts on Slack +* A IAM Role with EC2 ReadOnly access for EC2 instances monitoring. Only required for monitoring AWS nodes that are not part of the kubernetes cluster + + + +## Pre-Deployment + +Clone repository + + git clone github.com/camilb/prometheus-kubernetes && cd prometehus-kubernetes/basic + +Make any desired configuration changes in `configmaps` according to your setup. +* ./k8s/prometheus/prometheus.cm.yaml +* ./k8s/prometheus/alertmanager.cm.yaml + + +## Deploy Prometheus, Alertmaneger, Node Exporter, Grafana and Kube State Metrics + + ./init.sh + +* The init script will ask some basic questions and attempt to auto-discover information about your system. + + +Now you can access the dashboards locally using `kubectl port-forward`command, creating a ingress or a LoadBalancer. Please check the `./tools` directory to quickly configure a ingress or proxy the services to localhost. + +To remove everything, just execute the `./remove.sh` script. diff --git a/cleanup.sh b/basic/cleanup.sh similarity index 100% rename from cleanup.sh rename to basic/cleanup.sh diff --git a/init.sh b/basic/deploy similarity index 99% rename from init.sh rename to basic/deploy index 917cbde..c5395fc 100755 --- a/init.sh +++ b/basic/deploy @@ -1,8 +1,8 @@ #!/bin/bash #AWS_DEFAULT_AVAILABILITY_ZONE=us-east-1c -GRAFANA_DEFAULT_VERSION=4.5.1 +GRAFANA_DEFAULT_VERSION=4.5.2 PROMETHEUS_DEFAULT_VERSION=v2.0.0-beta.5 -ALERT_MANAGER_DEFAULT_VERSION=v0.8.0 +ALERT_MANAGER_DEFAULT_VERSION=v0.9.1 NODE_EXPORTER_DEFAULT_VERSION=v0.14.0 KUBE_STATE_METRICS_DEFAULT_VERSION=v1.0.1 DOCKER_REGISTRY_DEFAULT=docker.io diff --git a/grafana/Dockerfile b/basic/grafana/Dockerfile similarity index 100% rename from grafana/Dockerfile rename to basic/grafana/Dockerfile diff --git a/grafana/grafana-config/grafana.ini b/basic/grafana/grafana-config/grafana.ini similarity index 100% rename from grafana/grafana-config/grafana.ini rename to basic/grafana/grafana-config/grafana.ini diff --git a/grafana/grafana-dashboards/ec2-instances.json b/basic/grafana/grafana-dashboards/ec2-instances.json similarity index 100% rename from grafana/grafana-dashboards/ec2-instances.json rename to basic/grafana/grafana-dashboards/ec2-instances.json diff --git a/grafana/grafana-dashboards/k8s-cluster.json b/basic/grafana/grafana-dashboards/k8s-cluster.json similarity index 100% rename from grafana/grafana-dashboards/k8s-cluster.json rename to basic/grafana/grafana-dashboards/k8s-cluster.json diff --git a/grafana/grafana-dashboards/k8s-nodes.json b/basic/grafana/grafana-dashboards/k8s-nodes.json similarity index 100% rename from grafana/grafana-dashboards/k8s-nodes.json rename to basic/grafana/grafana-dashboards/k8s-nodes.json diff --git a/grafana/grafana-dashboards/prometheus-data-exploration.json b/basic/grafana/grafana-dashboards/prometheus-data-exploration.json similarity index 100% rename from grafana/grafana-dashboards/prometheus-data-exploration.json rename to basic/grafana/grafana-dashboards/prometheus-data-exploration.json diff --git a/k8s/grafana/grafana.svc.de.yaml b/basic/k8s/grafana/grafana.svc.de.yaml similarity index 100% rename from k8s/grafana/grafana.svc.de.yaml rename to basic/k8s/grafana/grafana.svc.de.yaml diff --git a/k8s/kube-state-metrics/ksm.de.yaml b/basic/k8s/kube-state-metrics/ksm.de.yaml similarity index 100% rename from k8s/kube-state-metrics/ksm.de.yaml rename to basic/k8s/kube-state-metrics/ksm.de.yaml diff --git a/k8s/kube-state-metrics/ksm.svc.yaml b/basic/k8s/kube-state-metrics/ksm.svc.yaml similarity index 100% rename from k8s/kube-state-metrics/ksm.svc.yaml rename to basic/k8s/kube-state-metrics/ksm.svc.yaml diff --git a/k8s/prometheus/alertmanager.cm.yaml b/basic/k8s/prometheus/alertmanager.cm.yaml similarity index 100% rename from k8s/prometheus/alertmanager.cm.yaml rename to basic/k8s/prometheus/alertmanager.cm.yaml diff --git a/k8s/prometheus/alertmanager.svc.de.yaml b/basic/k8s/prometheus/alertmanager.svc.de.yaml similarity index 100% rename from k8s/prometheus/alertmanager.svc.de.yaml rename to basic/k8s/prometheus/alertmanager.svc.de.yaml diff --git a/k8s/prometheus/alerts.cm.yaml b/basic/k8s/prometheus/alerts.cm.yaml similarity index 100% rename from k8s/prometheus/alerts.cm.yaml rename to basic/k8s/prometheus/alerts.cm.yaml diff --git a/k8s/prometheus/node-exporter.svc.ds.yaml b/basic/k8s/prometheus/node-exporter.svc.ds.yaml similarity index 100% rename from k8s/prometheus/node-exporter.svc.ds.yaml rename to basic/k8s/prometheus/node-exporter.svc.ds.yaml diff --git a/k8s/prometheus/prometheus.cm.yaml b/basic/k8s/prometheus/prometheus.cm.yaml similarity index 100% rename from k8s/prometheus/prometheus.cm.yaml rename to basic/k8s/prometheus/prometheus.cm.yaml diff --git a/k8s/prometheus/prometheus.svc.ss.yaml b/basic/k8s/prometheus/prometheus.svc.ss.yaml similarity index 100% rename from k8s/prometheus/prometheus.svc.ss.yaml rename to basic/k8s/prometheus/prometheus.svc.ss.yaml diff --git a/k8s/rbac/kube-state-metrics-rbac.yaml b/basic/k8s/rbac/kube-state-metrics-rbac.yaml similarity index 100% rename from k8s/rbac/kube-state-metrics-rbac.yaml rename to basic/k8s/rbac/kube-state-metrics-rbac.yaml diff --git a/k8s/rbac/prometheus-rbac.yaml b/basic/k8s/rbac/prometheus-rbac.yaml similarity index 100% rename from k8s/rbac/prometheus-rbac.yaml rename to basic/k8s/rbac/prometheus-rbac.yaml diff --git a/remove.sh b/basic/teardown similarity index 88% rename from remove.sh rename to basic/teardown index 8229d00..5f3a473 100755 --- a/remove.sh +++ b/basic/teardown @@ -5,4 +5,3 @@ kubectl delete -R -f ./k8s/prometheus kubectl delete -f ./k8s/kube-state-metrics kubectl delete -f ./k8s/rbac/prometheus-rbac.yaml kubectl delete -f ./k8s/rbac/kube-state-metrics-rbac.yaml -kubectl delete ns monitoring diff --git a/operator/tools/alertmanager_proxy.sh b/basic/tools/alertmanager_proxy.sh similarity index 83% rename from operator/tools/alertmanager_proxy.sh rename to basic/tools/alertmanager_proxy.sh index aad1cdc..226c14b 100755 --- a/operator/tools/alertmanager_proxy.sh +++ b/basic/tools/alertmanager_proxy.sh @@ -1,4 +1,4 @@ #!/bin/bash -POD=$(kubectl get pods --namespace=monitoring | grep alertmanager-main-0| cut -d ' ' -f 1) +POD=$(kubectl get pods --namespace=monitoring | grep alertmanager| cut -d ' ' -f 1) kubectl port-forward $POD --namespace=monitoring 9093:9093 diff --git a/operator/tools/grafana_proxy.sh b/basic/tools/grafana_proxy.sh similarity index 100% rename from operator/tools/grafana_proxy.sh rename to basic/tools/grafana_proxy.sh diff --git a/operator/tools/ingress/basic-auth.secret.yaml b/basic/tools/ingress/basic-auth.secret.yaml similarity index 100% rename from operator/tools/ingress/basic-auth.secret.yaml rename to basic/tools/ingress/basic-auth.secret.yaml diff --git a/operator/tools/ingress/ingress.yaml b/basic/tools/ingress/ingress.yaml similarity index 89% rename from operator/tools/ingress/ingress.yaml rename to basic/tools/ingress/ingress.yaml index a57d213..da8d2c7 100644 --- a/operator/tools/ingress/ingress.yaml +++ b/basic/tools/ingress/ingress.yaml @@ -14,14 +14,14 @@ spec: paths: - path: / backend: - serviceName: prometheus-k8s + serviceName: prometheus servicePort: 9090 - host: alertmanager.domain_name http: paths: - path: / backend: - serviceName: alertmanager-main + serviceName: alertmanager servicePort: 9093 - host: grafana.domain_name http: diff --git a/operator/tools/ingress/init.sh b/basic/tools/ingress/init.sh old mode 100755 new mode 100644 similarity index 100% rename from operator/tools/ingress/init.sh rename to basic/tools/ingress/init.sh diff --git a/operator/tools/ingress/nginx-controller.yaml b/basic/tools/ingress/nginx-controller.yaml similarity index 100% rename from operator/tools/ingress/nginx-controller.yaml rename to basic/tools/ingress/nginx-controller.yaml diff --git a/operator/tools/ingress/rbac.yaml b/basic/tools/ingress/rbac.yaml similarity index 74% rename from operator/tools/ingress/rbac.yaml rename to basic/tools/ingress/rbac.yaml index 796220d..7748038 100644 --- a/operator/tools/ingress/rbac.yaml +++ b/basic/tools/ingress/rbac.yaml @@ -14,12 +14,6 @@ rules: verbs: - list - watch - - apiGroups: - - "" - resources: - - nodes - verbs: - - get - apiGroups: - "" resources: @@ -42,7 +36,6 @@ rules: - events verbs: - create - - patch - apiGroups: - "extensions" resources: @@ -62,28 +55,8 @@ rules: - configmaps - pods - secrets - - namespaces verbs: - get - - apiGroups: - - "" - resources: - - configmaps - resourceNames: - # Defaults to "-" - # Here: "-" - # This has to be adapted if you change either parameter - # when launching the nginx-ingress-controller. - - "ingress-controller-leader-nginx" - verbs: - - get - - update - - apiGroups: - - "" - resources: - - configmaps - verbs: - - create - apiGroups: - "" resources: diff --git a/basic/tools/prometheus_proxy.sh b/basic/tools/prometheus_proxy.sh new file mode 100755 index 0000000..53e6f6f --- /dev/null +++ b/basic/tools/prometheus_proxy.sh @@ -0,0 +1,4 @@ +#!/bin/bash + +POD=$(kubectl get pods --namespace=monitoring | grep prometheus| cut -d ' ' -f 1) +kubectl port-forward $POD --namespace=monitoring 9090:9090 diff --git a/operator/deploy b/deploy similarity index 100% rename from operator/deploy rename to deploy diff --git a/operator/manifests/alertmanager/alertmanager.cm.yaml b/manifests/alertmanager/alertmanager.cm.yaml similarity index 100% rename from operator/manifests/alertmanager/alertmanager.cm.yaml rename to manifests/alertmanager/alertmanager.cm.yaml diff --git a/operator/manifests/alertmanager/alertmanager.svc.yaml b/manifests/alertmanager/alertmanager.svc.yaml similarity index 100% rename from operator/manifests/alertmanager/alertmanager.svc.yaml rename to manifests/alertmanager/alertmanager.svc.yaml diff --git a/operator/manifests/alertmanager/alertmanager.yaml b/manifests/alertmanager/alertmanager.yaml similarity index 100% rename from operator/manifests/alertmanager/alertmanager.yaml rename to manifests/alertmanager/alertmanager.yaml diff --git a/operator/manifests/grafana/grafana-dashboards.cm.yaml b/manifests/grafana/grafana-dashboards.cm.yaml similarity index 100% rename from operator/manifests/grafana/grafana-dashboards.cm.yaml rename to manifests/grafana/grafana-dashboards.cm.yaml diff --git a/operator/manifests/grafana/grafana.de.yaml b/manifests/grafana/grafana.de.yaml similarity index 100% rename from operator/manifests/grafana/grafana.de.yaml rename to manifests/grafana/grafana.de.yaml diff --git a/operator/manifests/grafana/grafana.svc.yaml b/manifests/grafana/grafana.svc.yaml similarity index 100% rename from operator/manifests/grafana/grafana.svc.yaml rename to manifests/grafana/grafana.svc.yaml diff --git a/operator/manifests/k8s/self-hosted/kube-controller-manager.yaml b/manifests/k8s/self-hosted/kube-controller-manager.yaml similarity index 100% rename from operator/manifests/k8s/self-hosted/kube-controller-manager.yaml rename to manifests/k8s/self-hosted/kube-controller-manager.yaml diff --git a/operator/manifests/k8s/self-hosted/kube-dns.yaml b/manifests/k8s/self-hosted/kube-dns.yaml similarity index 100% rename from operator/manifests/k8s/self-hosted/kube-dns.yaml rename to manifests/k8s/self-hosted/kube-dns.yaml diff --git a/operator/manifests/k8s/self-hosted/kube-scheduler.yaml b/manifests/k8s/self-hosted/kube-scheduler.yaml similarity index 100% rename from operator/manifests/k8s/self-hosted/kube-scheduler.yaml rename to manifests/k8s/self-hosted/kube-scheduler.yaml diff --git a/operator/manifests/kube-state-metrics/kube-state-metrics-rbac.yaml b/manifests/kube-state-metrics/kube-state-metrics-rbac.yaml similarity index 100% rename from operator/manifests/kube-state-metrics/kube-state-metrics-rbac.yaml rename to manifests/kube-state-metrics/kube-state-metrics-rbac.yaml diff --git a/operator/manifests/kube-state-metrics/kube-state-metrics.de.yaml b/manifests/kube-state-metrics/kube-state-metrics.de.yaml similarity index 100% rename from operator/manifests/kube-state-metrics/kube-state-metrics.de.yaml rename to manifests/kube-state-metrics/kube-state-metrics.de.yaml diff --git a/operator/manifests/kube-state-metrics/kube-state-metrics.svc.yaml b/manifests/kube-state-metrics/kube-state-metrics.svc.yaml similarity index 100% rename from operator/manifests/kube-state-metrics/kube-state-metrics.svc.yaml rename to manifests/kube-state-metrics/kube-state-metrics.svc.yaml diff --git a/operator/manifests/node-exporter/node-exporter.ds.yaml b/manifests/node-exporter/node-exporter.ds.yaml similarity index 100% rename from operator/manifests/node-exporter/node-exporter.ds.yaml rename to manifests/node-exporter/node-exporter.ds.yaml diff --git a/operator/manifests/node-exporter/node-exporter.svc.yaml b/manifests/node-exporter/node-exporter.svc.yaml similarity index 100% rename from operator/manifests/node-exporter/node-exporter.svc.yaml rename to manifests/node-exporter/node-exporter.svc.yaml diff --git a/operator/manifests/prometheus-operator/prometheus-operator-cluster-role-binding.yaml b/manifests/prometheus-operator/prometheus-operator-cluster-role-binding.yaml similarity index 100% rename from operator/manifests/prometheus-operator/prometheus-operator-cluster-role-binding.yaml rename to manifests/prometheus-operator/prometheus-operator-cluster-role-binding.yaml diff --git a/operator/manifests/prometheus-operator/prometheus-operator-service.yaml b/manifests/prometheus-operator/prometheus-operator-service.yaml similarity index 100% rename from operator/manifests/prometheus-operator/prometheus-operator-service.yaml rename to manifests/prometheus-operator/prometheus-operator-service.yaml diff --git a/operator/manifests/prometheus-operator/prometheus-operator.yaml b/manifests/prometheus-operator/prometheus-operator.yaml similarity index 100% rename from operator/manifests/prometheus-operator/prometheus-operator.yaml rename to manifests/prometheus-operator/prometheus-operator.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-rbac.yaml b/manifests/prometheus/prometheus-k8s-rbac.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-rbac.yaml rename to manifests/prometheus/prometheus-k8s-rbac.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-rules.yaml b/manifests/prometheus/prometheus-k8s-rules.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-rules.yaml rename to manifests/prometheus/prometheus-k8s-rules.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-alertmanager.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-alertmanager.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-alertmanager.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-alertmanager.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-apiserver.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-apiserver.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-apiserver.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-apiserver.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-controller-manager.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-kube-controller-manager.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-controller-manager.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-kube-controller-manager.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-dns.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-kube-dns.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-dns.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-kube-dns.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-scheduler.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-kube-scheduler.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-scheduler.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-kube-scheduler.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-state-metrics.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-kube-state-metrics.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-kube-state-metrics.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-kube-state-metrics.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-kubelet.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-node-exporter.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-node-exporter.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-node-exporter.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-node-exporter.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-prometheus-operator.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-prometheus-operator.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-prometheus-operator.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-prometheus-operator.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service-monitor-prometheus.yaml b/manifests/prometheus/prometheus-k8s-service-monitor-prometheus.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service-monitor-prometheus.yaml rename to manifests/prometheus/prometheus-k8s-service-monitor-prometheus.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s-service.yaml b/manifests/prometheus/prometheus-k8s-service.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s-service.yaml rename to manifests/prometheus/prometheus-k8s-service.yaml diff --git a/operator/manifests/prometheus/prometheus-k8s.yaml b/manifests/prometheus/prometheus-k8s.yaml similarity index 100% rename from operator/manifests/prometheus/prometheus-k8s.yaml rename to manifests/prometheus/prometheus-k8s.yaml diff --git a/operator/README.md b/operator/README.md deleted file mode 100644 index 6a7fd14..0000000 --- a/operator/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# Monitoring Kubernetes clusters on AWS using Prometheus Operator by CoreOS - - -![alt](https://www.camil.org/content/images/2017/cluster.png) - -Note: the work on this directory is based on CoreOs's [kube-prometheus](https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus). -The purpose of this project is to provide a simple and interactive method to deploy Prometheus Operator. - -## Features -* Prometheus Operator with support for Prometheus v2.X.X -* highly available Prometheus and Alermaneger -* InCluster deployment using `StatefulSets` for persistent storage -* auto-discovery for services and pods -* automatic RBAC configuration -* preconfigured alerts -* preconfigured Grafana dashboards -* easy to setup; usually less than a minute to deploy a basic monitoring solution for Kubernetes -* support for Kubernetes v1.7.x and up - -## One minute deployment - -[![asciicast](https://asciinema.org/a/139033.png)](https://asciinema.org/a/139033) - -## Prerequisites - -* Kubernetes cluster and `kubectl` configured -* Security Groups configured to allow the fallowing ports: - * 9100/TCP - node-exporter - * 10250/TCP - kubernetes nodes metrics, - * 10251/TCP - kube-scheduler - * 10252/TCP - kube-controller-manager - * 10054/TCP and 10055/TCP - kube-dns - -#### Optional -* SMTP Account for email alerts -* Token for alerts on Slack - -## Pre-Deployment - -Clone repository - - git clone github.com/camilb/prometheus-kubernetes && cd prometehus-kubernetes - -Make any desired configuration changes in `./assets` according to your setup. -You can also change the number of replicas for prometheus (default: 2) and alertmanager (default: 3) - - -## Deploy - - ./deploy - -* The deploy script will ask some basic questions and attempt to auto-discover information about your system. - - -Now you can access the dashboards locally using `kubectl port-forward`command, creating a ingress or a LoadBalancer. Please check the `./tools` directory to quickly configure a ingress or proxy the services to localhost. - -To remove everything, just execute the `./teardown` script. diff --git a/operator/scripts/generate-grafana-credentials-secret.sh b/operator/scripts/generate-grafana-credentials-secret.sh deleted file mode 100755 index 3244922..0000000 --- a/operator/scripts/generate-grafana-credentials-secret.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash - -if [ "$#" -ne 2 ]; then - echo "Usage: $0 user password" - exit 1 -fi - -os=$(uname) -if [[ $os == "Darwin" ]]; then - b64="base64" -else - b64="base64 --wrap=0" -fi - -user=$1 -password=$2 - -cat <<-EOF -apiVersion: v1 -kind: Secret -metadata: - name: grafana-credentials -data: - user: $(echo -n ${user} | $b64) - password: $(echo -n ${password} | $b64) -EOF diff --git a/operator/scripts/generate-manifests.sh b/operator/scripts/generate-manifests.sh deleted file mode 100755 index bb0c068..0000000 --- a/operator/scripts/generate-manifests.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash - -# Generate Alert Rules ConfigMap -hack/scripts/generate-rules-configmap.sh > manifests/prometheus/prometheus-k8s-rules.yaml - -# Generate Dashboard ConfigMap -#hack/scripts/generate-dashboards-configmap.sh > manifests/grafana/grafana-dashboards.yaml - -# Generate Dashboard ConfigMap with configmap-generator tool -# Max Size per ConfigMap: 240000 -# Input dir: assets/grafana -# output file: manifests/grafana/grafana-dashboards.yaml -# grafana deployment output file: manifests/grafana/grafana-deployment.yaml -test -f manifests/grafana/grafana-dashboards.yaml && rm -f manifests/grafana/grafana-dashboards.yaml -test -f manifests/grafana/grafana-deployment.yaml && rm -f manifests/grafana/grafana-deployment.yaml -hack/grafana-dashboards-configmap-generator/bin/grafana_dashboards_generate.sh -s 240000 -i assets/grafana -o manifests/grafana/grafana-dashboards.yaml -g manifests/grafana/grafana-deployment.yaml - -# Generate Grafana Credentials Secret -hack/scripts/generate-grafana-credentials-secret.sh admin admin > manifests/grafana/grafana-credentials.yaml - -# Generate Secret for Alertmanager config -hack/scripts/generate-alertmanager-config-secret.sh > manifests/alertmanager/alertmanager-config.yaml - diff --git a/operator/tools/prometheus_proxy.sh b/operator/tools/prometheus_proxy.sh deleted file mode 100755 index 068c15c..0000000 --- a/operator/tools/prometheus_proxy.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/bash - -POD=$(kubectl get pods --namespace=monitoring | grep prometheus-k8s-0| cut -d ' ' -f 1) -kubectl port-forward $POD --namespace=monitoring 9090:9090 diff --git a/release/download b/release/download new file mode 100755 index 0000000..915d421 --- /dev/null +++ b/release/download @@ -0,0 +1,18 @@ +#! /bin/sh + +PROMKUBE_VERSION=${PROMKUBE_VERSION:-v2.0.0-beta.3} + +NAME="prometheus-kubernetes" + +URL="https://github.com/camilb/$NAME" +echo "Clonning $NAME from $URL ..." +tput sgr0 + +git clone "$URL" +cd $NAME +echo + +git -c advice.detachedHead=false checkout tags/$PROMKUBE_VERSION -b $NAME +tput sgr0 +echo +echo "Go to ./prometheus-kubernetes and execute ./deploy to install HA Prometheus Operator." diff --git a/operator/scripts/generate-alertmanager-config-secret.sh b/scripts/generate-alertmanager-config-secret.sh similarity index 100% rename from operator/scripts/generate-alertmanager-config-secret.sh rename to scripts/generate-alertmanager-config-secret.sh diff --git a/operator/scripts/generate-dashboards-configmap.sh b/scripts/generate-dashboards-configmap.sh similarity index 100% rename from operator/scripts/generate-dashboards-configmap.sh rename to scripts/generate-dashboards-configmap.sh diff --git a/operator/scripts/generate-rules-configmap.sh b/scripts/generate-rules-configmap.sh similarity index 100% rename from operator/scripts/generate-rules-configmap.sh rename to scripts/generate-rules-configmap.sh diff --git a/operator/scripts/wrap-dashboard.sh b/scripts/wrap-dashboard.sh similarity index 100% rename from operator/scripts/wrap-dashboard.sh rename to scripts/wrap-dashboard.sh diff --git a/operator/teardown b/teardown similarity index 100% rename from operator/teardown rename to teardown diff --git a/tools/alertmanager_proxy.sh b/tools/alertmanager_proxy.sh index 226c14b..aad1cdc 100755 --- a/tools/alertmanager_proxy.sh +++ b/tools/alertmanager_proxy.sh @@ -1,4 +1,4 @@ #!/bin/bash -POD=$(kubectl get pods --namespace=monitoring | grep alertmanager| cut -d ' ' -f 1) +POD=$(kubectl get pods --namespace=monitoring | grep alertmanager-main-0| cut -d ' ' -f 1) kubectl port-forward $POD --namespace=monitoring 9093:9093 diff --git a/tools/ingress/ingress.yaml b/tools/ingress/ingress.yaml index da8d2c7..a57d213 100644 --- a/tools/ingress/ingress.yaml +++ b/tools/ingress/ingress.yaml @@ -14,14 +14,14 @@ spec: paths: - path: / backend: - serviceName: prometheus + serviceName: prometheus-k8s servicePort: 9090 - host: alertmanager.domain_name http: paths: - path: / backend: - serviceName: alertmanager + serviceName: alertmanager-main servicePort: 9093 - host: grafana.domain_name http: diff --git a/tools/ingress/init.sh b/tools/ingress/init.sh old mode 100644 new mode 100755 diff --git a/tools/ingress/rbac.yaml b/tools/ingress/rbac.yaml index 7748038..796220d 100644 --- a/tools/ingress/rbac.yaml +++ b/tools/ingress/rbac.yaml @@ -14,6 +14,12 @@ rules: verbs: - list - watch + - apiGroups: + - "" + resources: + - nodes + verbs: + - get - apiGroups: - "" resources: @@ -36,6 +42,7 @@ rules: - events verbs: - create + - patch - apiGroups: - "extensions" resources: @@ -55,8 +62,28 @@ rules: - configmaps - pods - secrets + - namespaces verbs: - get + - apiGroups: + - "" + resources: + - configmaps + resourceNames: + # Defaults to "-" + # Here: "-" + # This has to be adapted if you change either parameter + # when launching the nginx-ingress-controller. + - "ingress-controller-leader-nginx" + verbs: + - get + - update + - apiGroups: + - "" + resources: + - configmaps + verbs: + - create - apiGroups: - "" resources: diff --git a/tools/prometheus_proxy.sh b/tools/prometheus_proxy.sh index 53e6f6f..068c15c 100755 --- a/tools/prometheus_proxy.sh +++ b/tools/prometheus_proxy.sh @@ -1,4 +1,4 @@ #!/bin/bash -POD=$(kubectl get pods --namespace=monitoring | grep prometheus| cut -d ' ' -f 1) +POD=$(kubectl get pods --namespace=monitoring | grep prometheus-k8s-0| cut -d ' ' -f 1) kubectl port-forward $POD --namespace=monitoring 9090:9090