DSE Metrics Collector Dashboards

This repository contains preconfigured Grafana dashboards that integrate with DSE Metrics Collector. Use DSE Metrics Collector to export DSE metrics to a monitoring tool like Prometheus, and then visualize DSE metrics in the Grafana dashboards.

Use Docker and modify the provided Prometheus configuration file, or manually export DSE metrics to an existing Prometheus server. Although the examples in the linked documentation use Prometheus as the monitoring tool, you can export the aggregated metrics to other tools like Graphite and Splunk.

WARNING

By default, Prometheus is configured to retain 15 days' worth of data only. To avoid unexpected loss of data metrics, adjust the default rentention period. To keep a history of your metrics data beyond the default setting, you must add the --storage.tsdb.retention= flag to the docker-compose.yml file at the Prometheus runtime command line.

Getting started

Clone this repository and then follow the instructions in the DataStax documentation based on your implementation:

• Export and visualize metrics with Prometheus and Docker;
• Manually export and visualize metrics with Prometheus.

Generation of the tg_dse.json file

In some cases, such as using a big cluster, the tg_dse.json file could be generated by one of the auxiliary scripts located in the extras directory. These scripts are examples, and do not handle every configuration, such as authentication:

• generate-discovery-file.sh: this script uses the nodetool command to extract a list of servers in the cluster, and generate a service discovery file. Run this script on a single nodes in the cluster. The script accepts the following parameters:
  • Name of file to which the data will be written (Required)
  • Port number (Optional, defaults to 9103)
• generate-discovery-file.py: this script uses the DSE Python driver to fetch cluster metadata, and generate a service discovery file. The script accepts the following parameters:
  • Contact point that is used to connect to cluster (Required)
  • Name of file to which the data will be written (Required)
  • Port number (Optional, defaults to 9103)

These scripts can also be used for periodic refresh of the tg_dse.json file to reflect changes in the cluster's topology.

Support

The code, examples, and snippets provided in this repository are not "Supported Software" under any DataStax subscriptions or other agreements.

Slack - https://academy.datastax.com/slack #dse-metric-collector

License

Please refer to LICENSE file.

Examples

The following screenshots illustrate the preconfigured dashboards in this repository.

DSE Cluster Condensed

DSE System & Node Metrics

DSE Cluster Latest

Prometheus Statistics

Kubernetes Support

These dashboards and prometheus configuration files may be transformed via the collection of Python scripts under bin/.

Specifically run bin/clean.py && bin/build.py to generate appropriate files in the dist directory. These files integrate with the Custom Resources defined by the Prometheus and Grafana operators available on Operator Hub.

Prometheus

The Prometheus Operator handles the orchestration, configuration, and deployment of k8s resources required for a HA Prometheus installation. Rather than specifying a list of DSE nodes in a JSON file we direct Prometheus to monitor a Kubernetes Service which exposes all DSE nodes via DNS. This mapping of hosts is handled by a ServiceMonitor Custom Resource defined by the operator. The following steps illustrate how to install the Prometheus operator, deploy a service monitor pointed at a DSE cluster (with metric relabeling), and deploy a HA prometheus cluster connected to the service monitor.

1. Install the OperatorHub Lifecycle Manager (OLM)

   curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.14.1/install.sh | bash -s 0.14.1

   This installs a number of custom resource definitions and an operator that handles installing other operators.

2. Install the Prometheus Operator

   kubectl create -f dist/prometheus/k8s/operator.yaml

   This will pull down and start the Prometheus operator. The operator is installed in a new namespace prometheus-operator along with a configuration indicating it should watch the default namespace for resources it manages. Should the installation require different namespaces change the values within this file and re-apply it to the cluster.

3. Configure and install the Service Monitor

   Before installation edit the service monitor's YAML to include the appropriate labels that match your cluster's service. For example if your cluster service has the label com.datastax.dse.cluster: dse-cluster this mapping would be included in the service monitor yaml under spec.selector.matchLabels. As a convenience we have included the example here in the generated service_monitor.yaml.

   With the configuration file updated apply the resource to the cluster.

   kubectl apply -f dist/prometheus/k8s/service_monitor.yaml

4. Configure and install the Prometheus deployment

   Similar to the service and service monitor mapping, the instance must be provided with a set of labels to match service monitors to Prometheus deployments. The following section in the Prometheus custom resource maps the deployment to all service monitors with the label monitoring=dse-cluster

     serviceMonitorSelector:
       matchLabels:
         monitoring: dse-cluster
   

   After adjustments apply this file to the cluster

   kubectl apply -f dist/prometheus/k8s/instance.yaml

Grafana

The Grafana Operator handles the orchestration, configuration, and deployment of k8s resources required for a Grafana installation. Instead of configuring datasources, dashboards, and deployments via the GUI everything is configured via YAML files in this repo. The following steps illustrate how to install the Grafana operator, data source pointed at the previously deployed prometheus, a collection of dashboards, and an instance of the grafana application.

1. Install the OperatorHub Lifecycle Manager (OLM) if you haven't already

   curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.14.1/install.sh | bash -s 0.14.1

   This installs a number of custom resource definitions and an operator that handles installing other operators.

2. Install the Grafana Operator

   kubectl create -f dist/grafana/k8s/operator.yaml

   This will pull down and start the Grafana operator. The operator is installed in a new namespace grafana-operator along with a configuration indicating it should watch the default namespace for resources it manages. Should the installation require different namespaces change the values within this file and re-apply it to the cluster.

3. Configure and install the GrafanaDataSource

   Before installation edit the YAML to point at the instantiated prometheus cluster. If you are not sure what value to use check the output of kubectl get svc for the prometheus service name.

   With the configuration file updated apply the resource to the cluster.

   kubectl apply -f dist/grafana/k8s/datasource.yaml

4. Configure and install the GrafanaDashboard

   Before installation edit the YAML with appropriate labels. In this example a label of app=grafana is used. With the configuration file updated apply the resource to the cluster.

   kubectl apply -f dist/grafana/k8s/*.dashboard.yaml

5. Configure and install the Grafana deployment

   The deployment must be informed of labels to check for matching dashboards. The following section in the Grafana custom resource maps the deployment to all dashboards with the label app=grafana

     dashboardLabelSelector:
       - matchExpressions:
           - key: app
             operator: In
             values:
               - grafana
   

   After adjustments apply this file to the cluster

   kubectl apply -f dist/grafana/k8s/instance.yaml