KubeCon EU 2020

This repo contains the source material for my demos for KubeCon EU 2020.

Simple Automated Thanos Receive Hashring

Our first exercise will be to deploy a Thanos receive hashring along with the Thanos receive controller. This simple yet effective stack will help us build a foundation for more advanced configurations later.

1. To start, let's create the thanos Namespace:

   kubectl create ns thanos

2. Create the resources in the directory for the first demo:

   kubectl apply -f 1

3. Let's take a look at the Pods we created:

   kubectl get pods -n thanos -o wide

4. Now, let's see the configuration that the Thanos receive controller generated:

   kubectl get configmap -n thanos thanos-receive-controller-tenants-generated -o jsonpath="{.data['hashrings\.json']}" | jq

5. Our Pods are running and the configuration was generated, so now let's test the 1-node hashring; in another terminal window run a process to make remote-write requests against the Thanos receive service:

   docker run --rm quay.io/observatorium/up --endpoint-type metrics --endpoint-write http://"$(kubectl get service -n thanos thanos-receive -o jsonpath="{.spec.clusterIP}")":19291/api/v1/receive --period=1s

6. Let's scale up the hashring to three replicas:

   kubectl -n thanos scale statefulset thanos-receive-default --replicas=3

7. Now let's check the Pods and the generated configuration:

   kubectl get pods -n thanos -o wide -w
   kubectl get configmap -n thanos thanos-receive-controller-tenants-generated -o jsonpath="{.data['hashrings\.json']}" | jq

Autoscaling Hashring

In this exercise, we'll attempt to deploy an autoscaling Thanos receive Hashring, where the hashring grows to accommodate increased request load.

Note: these instructions assume that the Kubernetes metrics-server, or something else implementing the v1beta1.metrics.k8s.io resource metrics API, e.g. the Prometheus Adapter, is installed on the cluster.

1. Begin by creating the thanos Namespace:

   kubectl create ns thanos

2. Next, let's deploy the resources from directory for the second demo:

   kubectl apply -f 2

3. While these are being created, let's take a look at the differences from the first demo. You should see that the Thanos receive Pod template now includes a sidecar called configmap-to-disk. We use this sidecar to automatically synchronize the hashring configuration file from the API whenever the ConfigMap changes, without having to wait for the eventually consistent Kubelet volume mount to be update. This allows the Pods in the hashring to react faster to scaling events. You should also see that we defined a HorizontalPodAutoscaler resource that scales the hashring based on CPU utilization This resource also clamps the minimum amount of replicas in the StatefulSet to 5, since we want to make sure we always have some ingestion capacity available.

   Note: in order to scale based on CPU utilization, we must set resource requests on the containers being scaled.

   $EDITOR 2/thanos-receive-default-statefulset.yaml 2/thanos-receive-default-hpa.yaml

4. Let's take a look at the Pods we created:

   kubectl get pods -n thanos -o wide

5. Again, if everything rolled out successfully, we should see that the Thanos receive controller generated successfully generated a hashring configuration:

   kubectl get configmap -n thanos thanos-receive-controller-tenants-generated -o jsonpath="{.data['hashrings\.json']}" | jq

6. Let's keep a process running to watch how our Pods are rolling out:

   kubectl get pods -n thanos -w

7. In a new terminal, let's watch the scaling resource for the hashring:

   kubectl get horizontalpodautoscalers.autoscaling -n thanos thanos-receive-default -w

8. Finally, in another terminal, let's generate some serious load to test our stack. This time around we'll use Fresh Track's Avalanche project to generate remote-write requests. This process will make 1000 remote-write requests where each request has 100k time series.

   Note: we are using a fork of the project that doesn't cause the process to exit when it encounters an error. This should serve as an indication of how we expect the exercise to go.

   Run the Avalanche container and point it at the Service IP:

   docker run --rm squat/avalanche:log-errors --remote-url http://"$(kubectl get service -n thanos thanos-receive -o jsonpath="{.spec.clusterIP}")":19291/api/v1/receive --remote-requests-count=1000 --metric-count 1000 --series-count 100