Wrong Loki service_name label value for Kubernetes pod logs

[cyrille-leclerc]

The service_name label introduced in Loki v3 is populated with an erroneous value for Kubernetes pod logs collected by the Grafana Kubernetes Monitoring Helm Chart being populated with the container label value.

A Kubernetes best practice is to capture the application/service name using the app.kubernetes.io/name Kubernetes label (see Kubernetes Documentation / ... / Recommended Labels).

Som users will also be interested in defining their custom Kubernetes Label to populate the Loki service_namelabel.

Loki documentation with regard to the service_name label

https://grafana.com/docs/loki/latest/setup/upgrade/#service_name-label

service_name label
Loki 3.0 will automatically assign a service_name label to all ingested logs by default. A service name is something required by Open Telemetry semantic conventions and is something Grafana Labs is building into our future user interface and query experiences.

Loki will attempt to create the service_name label by looking for the following labels in this order:

• service
• app
• application
• name
• app_kubernetes_io_name
• container
• container_name
• component
• workload
• job

If no label is found matching the list, a value of unknown_service is applied.

You can change this list by providing a list of labels to discover_service_name in the limits_config block.

[petewall]

I'm on board with this, but I could use some clarification. The list Loki provides is the list of Loki labels. How should that translate to Kubernetes labels?

Meaning is component the Kubernetes label component, or app.kubernetes.io/component?
Also, should the Helm chart set service_name, or just set the list of labels above and rely on Loki to set service_name?

[cyrille-leclerc]

Great question.

Mapping the Kubernetes label app.kubernetes.io/name to the Loki label app_kubernetes_io_name is compelling to me:

• Pros: looks very Kubernetes and Prometheus native and works with Loki:
  • It's a Kubernetes best practice to capture the application/service name using the app.kubernetes.io/name Kubernetes label (see Kubernetes Documentation / ... / Recommended Labels).
  • The Prometheus docs use the example of mapping the label app.kubernetes.io/name to app_kubernetes_io_name (see here)
  • Loki uses app_kubernetes_io_name in its algorithm to infer service_name
• Cons: the Loki label app_kubernetes_io_name will remain and it doesn't feel very Loki native

An interesting alternative is to map the Kubernetes label app.kubernetes.io/name to the Loki label app

• Pros: looks very Prometheus and Kubernetes native and works with Loki:
  • It's a Kubernetes best practice to capture the application/service name using the app.kubernetes.io/name Kubernetes label (see Kubernetes Documentation / ... / Recommended Labels).
  • The app label feels very Prometheus native
  • Loki uses app in its algorithm to infer `service_name``
• Cons: It's not self explanatory that the Loki label app is derived from the Kubernetes label app.kubernetes.io/name.

The other log label names that Loki use to infer service_name are less compelling to me:

• service_name refers to the Otel semantic conventions which is not suited for our K8s Pod logs which aren't Otel logs
• service, or application don't feel like the Prometheus ecosystem

WDYT?

[zeitlinger]

logic of the otel operator:

https://github.com/open-telemetry/opentelemetry-operator/blob/1d759b599e92bb99708cbc5726e7b81af3a4caff/pkg/instrumentation/sdk.go#L281-L287

https://github.com/open-telemetry/opentelemetry-operator/blob/1d759b599e92bb99708cbc5726e7b81af3a4caff/pkg/instrumentation/sdk.go#L397-L420

OTEL_SERVICE_NAME env variable
k8s.deployment.name
k8s.replicatset.name
k8s.statefulset.name
k8s.daemonset.name
k8s.conjob.name
k8s.job.name
k8s.pod.name
k8s container name from the pod spcs

[cyrille-leclerc]

@petewall once we have a preference on the K8s label/metadata/spec attribute we want ot map by default, I propose that we get feedback from some users to verify our enhancement would meet their expectation.

[cyrille-leclerc]

FYI @zeitlinger suggestion the OpenTelemetry Operator should support the app.kubernetes.io/name K8s label to set service.name in OTel auto instrumentations

• Add support for k8s labels such as app.kubernetes.io/name open-telemetry/opentelemetry-operator#3112