confusion with annotations makes instrumentation do nothing

[netikras]

Component(s)

auto-instrumentation

Describe the issue you're reporting

okay, I'm just starting to dip my toes in the OTEL waters, so there's a good chance I'm doing something wrong. However, I'm following instructions first, before going all improv. So I expect it to work..

I've installed the operator and I'm now following this: https://opentelemetry.io/docs/kubernetes/operator/automatic/
It says to create a stock collector and an instrumentation configuration. And so I did. Then I deployed a petstore app with instrumentation annotations

apiVersion: v1
kind: Pod 
metadata:
  creationTimestamp: null
  labels:
    run: petstore
  name: petstore
  namespace: jsdev
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
spec:
  containers:
  - image: swaggerapi/petstore
    name: petstore
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

et voila -- I see events saying that instrumentation is kicking in. Superb!!

Then I tried to move it to another namespace. It worked too!

Then I read in the docs that the operator uses a hierarchy of annotations: on pod level and on namespace level (and k8s services too? I'm a bit fuzzy on this part). So I slap instrumentation.opentelemetry.io/inject-java: "true" on my namespace and redeploy the pod. It worked.

Then I figured "hey, my actual app hosts several different components in their namespace: java and nodejs based". So I added another annotation to the namespace: instrumentation.opentelemetry.io/inject-nodejs: "true". Delete the petstore, redeploy, and.... nothing. kubectl -n jsdev get events | grep open shows no new events regd instrumentation.

So I delete the pod and redeploy, just to see if this is consistent. And again, no new events.

As soon as I remove the nodejs annotation from the namespace and redeploy the pod, I see 2 new "open.*" events recorded.

Even if I keep the nodejs annotation on namespace and annotate my perstore pod with java, it still doesn't work.

Is this intentional? Am I doing it wrong? Or was I wrong to assume that auto-instrumentation does not automatically pick the right instrumentation approach for the container, so I don't have an option to list all of them -- I have to choose the exact instr. mode?

So if I have a namespace full of java apps and only one nodejs-based component mixed in, does it mean I no longer have the luxury of using the namespace-annotation and only overriding the injection for that nodejs pod? Do I have to specify the annotation for each and every pod now?

[jaronoff97]

So i think this is where our support for multi-container instrumentation would come in handy, but it's not the complete solution. I think a better solution is probably what is being discussed in #2744. For now, yes you would need to add an annotation on each of your pods. I'm going to close this issue in favor of the above to maintain the discussion there, please add a comment there if you get a chance!