How to collect stdout and stderr logs in sidecar mode?

[edwardzjl]

Hi there,

Thanks a lot for this amazing stack, our team have been using grafana / loki / promtail stack on kubernetes for a while.

We have been using promtail in daemonset mode, which collects all logs in the k8s cluster.

However, we are running k8s in multi-tenant environment. Not all tenants have access (or want) to collect their logs to our grafana / loki dashboard.

I have been reading the new grafana-agent documents for some days, and am willing to migrate to the new stack. If I understand it correctly, I can implement a multi-tenant log collecting stack by deploying grafana-agent in sidecar mode.

However, I have some questions on how can I implement that.

• First, the installation document (https://grafana.com/docs/agent/latest/flow/setup/install/kubernetes/) explains how to deploy grafana-agent as a daemonset using helm, but I haven't found how to deploy grafana-agent as a sidecar.
• Second, in our current implementation (cluster-wide grafana / loki / promtail-daemonset), the pod logs to stdout, the container runtime (containerd) collects logs to files on the host machine, and the promtail daemonset collects logs from host path (/var/log/containers). However, if I deploy grafana-agent as sidecar, how can I collect the stdout logs?

Sorry for the newbee questions, and thanks in advance for any responses.

[rfratto]

The sidecar pattern isn't fully recommended because the overhead of each container can quickly add up and consume a lot of resources.

However, if you must use a sidecar pattern, there are two choices to collect logs from a container:

1. The container to collect logs from must pipe stdout/stderr logs to a file accessible by the sidecar container
2. A combination of the discovery.kubelet and loki.source.kubernetes components can be used to discover the other containers in the same pod and collect logs from it; this uses the Kubernetes API for collecting logs. discovery.kubernetes may be used in place of discovery.kubelet, but with discovery.kubernetes, many sidecar containers can cause too much load for the Kubernetes API.

[edwardzjl]

Thank you so much for the quick response, I will stay with the daemonset pattern.