Two Layer Agent for tail sampling

[EvertonSA]

Hi,

everywhere on the internet it says we might need a two layer otel setup for tail sampling.

the grafana agent (as deamonset) in cluster mode is the same?

Do I need to deploy 1 layer as deamonset, and do k8sattribute processor and otel loadbalancing to a second layer where I do tail sampling and service graph metrics?

I did deploy to production as 1 layer setup with tail sampling + service graph but the setup seems incorrect. I feel I have been misled by articles that did not go in depth about tail sampling strategies.

---

after a lot of thoughts, I think a two layer setup should be the default recommended way if you are doing tail sampling, as the chart deploy daemonset by default. Am I wrong? Should we also have documentation stating this is a must have setup if you are doing tail sampling?

[tpaschalis]

It all depends on the setup of your infrastructure.

Generally, I wouldn't recommend a Daemonset for traces collection. If you absolutely must eg. collect node_exporter metrics or Kubernetes pod logs from the same K8S workload, you might be looking to split things up in multiple workloads.

Right now, my 2c would be:
a) If you're using Grafana Agent in clustering mode for logs/metrics/profiles collection, look into a Deployment or a Statefulset (for the WAL directory).
b) If you're using Grafana Agent for traces with tail sampling, you can use a single-tier Deployment,
c) If you reach a really high scale, and you need to start using the loadbalancing exporter, then use a double-tier setup where the first one does k8sattribute processor and loadbalancing while the second one does tail sampling and service graph metrics.

[EvertonSA]

amazing thanks. although we are not truly high scale, I decided to go with approach C.

First layer with Daemonsets collecting logs and sending to loki + otel collector with k8sattribute processor and loadbalancing
Second layer as StatefulSet (could be deployment also, idk/idc) doing Tail Sampling + service Graph.

---

on the applications, I'm sending the otel data to the closest collector, to the deamonset on the same node. For that, I have added an additional NodePort service and on the apps, I'm configuring it like this:

see kubernetes/ingress-nginx#9800 (comment)

controller:
  extraEnvs:
    - name: "OPENTELEMETRY_COLLECTOR_IP"
      valueFrom:
        fieldRef:
          fieldPath: "status.hostIP"
    - name: "OTEL_EXPORTER_OTLP_ENDPOINT"
      value: "http://$(OPENTELEMETRY_COLLECTOR_IP):31317"

maybe my setup is overkill for my usecase, but let me experiment for a few weeks like this

[tpaschalis]

Sounds good! We're looking for ways to improve this, so we'd love to hear back about how it worked out for you using Flow mode!