No sidecar Containers created for static pods, e.g. kube-apiserver.

[Jeansen]

Component(s)

collector, auto-instrumentation

What happened?

Description

No sidecar Containers created for static pods, e.g. kube-apiserver.

Steps to Reproduce

Create a custom resource, with mode set to sidecar in namespace kube-system. Add relevant annotation to this namespace.

Here's my current playground CR:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector-deployment
  namespace: "kube-system"
spec:
# tolerations:
#   - key: CriticalAddonsOnly
#     operator: Exists
#   - operator: Exists
#     effect: NoSchedule
#   - operator: Exists
#     effect: NoExecute
  mode: sidecar
  config: |
    receivers:
    # Make sure to add the otlp receiver.
    # This will open up the receiver on port 4317
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 50
        spike_limit_percentage: 30
      batch:
    extensions:
      health_check:
    exporters:
      debug:
      otlp/jaeger:
        endpoint: "jaeger-collector.observability.svc.cluster.local:4317"
        tls:
          insecure: true
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp/jaeger]
#       metrics:
#         receivers: [otlp]
#         processors: [memory_limiter]
#         exporters: [debug]

And here's what my kube-system NS looks like:

apiVersion: v1
kind: Namespace
metadata:
  name: kube-system
  uid: ee6a7062-5c71-43df-a639-04f1750e4012
  resourceVersion: '5104269'
  creationTimestamp: '2024-04-29T19:48:04Z'
  labels:
    k8slens-edit-resource-version: v1
    kubernetes.io/metadata.name: kube-system
  annotations:
    sidecar.opentelemetry.io/inject: "true"
  managedFields:
    - manager: kube-apiserver
      operation: Update
      apiVersion: v1
      time: '2024-04-29T19:48:04Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:kubernetes.io/metadata.name: {}
    - manager: node-fetch
      operation: Update
      apiVersion: v1
      time: '2024-05-11T18:33:19Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            f:k8slens-edit-resource-version: {}
  selfLink: /api/v1/namespaces/kube-system
status:
  phase: Active
spec:
  finalizers:
    - kubernetes

Expected Result

When deleting a kube-system pod, e.g. kube-apiserver, the new Pod should have a sidecar container.

Actual Result

There is no sidecar container. BUT: If I create a simple dummy deployment in the kube-system namespace, then a sidecar container will be created.

Kubernetes Version

1.30.0

Operator version

0.56.0

Collector version

0.56.0

Environment information

Environment

Nodes run on Debian 12 as VMs (QEMU/KVM). I have 3 masters, 4 workers. Container runtime is CRI-O 1.30.0

Log output

No response

Additional context

I am following along https://kubernetes.io/docs/concepts/cluster-administration/system-traces/ but without the sidecar, I had to resort to a deployment.

[pavolloffay]

@Jeansen the collector CR (the first code snippet) should have mode set to sidecar

[Jeansen]

@pavolloffay Oh, sorry. That is a typ or better copy/paste error after I had tried different settings. I updated the code accordingly.

[pavolloffay]

@frzifus can you help here? IIRC you looked at core k8s components, did you try to inject sidecar there?

[frzifus]

yes, I think it depends a bit on how you setup your k8s cluster. If for example your api-server and another component would run with hostNetwork: true there would be a conflict with the sidecars.

Instead we deploy a daemonset running with hostNetwork: true. That way, etcd, kubelet and the apiserver can report to the collector on each node.

Having a sperate CR could make sense. But tbh I did not think much about it.

Could you provide an example of what you would like to see @Jeansen ?

---

Kubecon tutorial [link].

FTR:

• Proposal: Opinionated Operator CRDs #1477

[Jeansen]

@frzifus Well, regarding e.g. the apiserver, I'd expect it to behave the same way as any other service when using sidecars. But as I wrote, it does not with respect to static Pods. So I tried using the Deployment mode. First, it did not work either because static Pods cannot use the cluster DNS. I reached out to the Kubernetes architects and they wrote:

Control plane components run with host networking; because of what these components do, they need to be able to boot before there is a healthy control plane for your cluster.

So, I created an ingress for the collector and with an external DNS it now works, at least for me (without sidecars).

Her's what my config looks like for the apiserver:

apiVersion: apiserver.config.k8s.io/v1beta1
kind: TracingConfiguration
#endpoint: "10.98.30.28:4317"
endpoint: "otel.k8s.lab:80"
samplingRatePerMillion: 1000000

Of course, as seen above in the comment, one could also use a specific IP of the collector service, but that's hardly what we want!

And here's the Ingress manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: otel-ingress
  namespace: kube-system
spec:
  ingressClassName: nginx
  rules:
  - host: otel.k8s.lab
    http:
      paths:
      - backend:
          service:
            name: otel-kube-system-collector
            port:
              number: 4317
        path: /
        pathType: Prefix

Finally, the apiserver needs to have --tracing-config-file set accordingly.

That is my current solution on my "way to Rome". If sidecars would work, all this would not be necessary. Even more, the TracingConfiguration would be quite generic using localhost instead of a fixed DNS name.