Operator not injecting specified Go auto instrumentation image

[stephenhoran]

Component(s)

auto-instrumentation

What happened?

Description

It appears the Operator is currently not injecting the specified Go Instrumentation image into containers that are configured for auto instrumentation while our python images seem to inject without issue. We have testing both version 0.104.0 and 0.103.0 of the operator with the same issue.

We have the operator deployed with helm and configured as such (please note the image registry has been edited to not leak information) :

go: repository: "{some container registry}/open-telemetry/autoinstrumentation-go" tag: "v0.14.0-alpha"

When the Operator is started, the Operator validates the configuration correctly with the appropriate image:

{"level":"INFO","timestamp":"2024-07-25T19:37:03Z","message":"Starting the OpenTelemetry Operator","opentelemetry-operator":"0.104.0","opentelemetry-collector":"{some container registry}/otel/opentelemetry-collector:0.103.1","opentelemetry-targetallocator":"ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.103.0","operator-opamp-bridge":"ghcr.io/open-telemetry/opentelemetry-operator/operator-opamp-bridge:0.103.0","auto-instrumentation-java":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.32.1","auto-instrumentation-nodejs":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.51.0","auto-instrumentation-python":"{some image repository}/open-telemetry/autoinstrumentation-python:latest","auto-instrumentation-dotnet":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:1.2.0","auto-instrumentation-go":"{some image repository/open-telemetry/autoinstrumentation-go:v0.14.0-alpha","auto-instrumentation-apache-httpd":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","auto-instrumentation-nginx":"ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.4","feature-gates":"-operator.golang.flags,operator.observability.prometheus","build-date":"2024-06-25T18:35:30Z","go-version":"go1.22.4","go-arch":"amd64","go-os":"linux","labels-filter":[],"annotations-filter":[],"enable-multi-instrumentation":false,"enable-apache-httpd-instrumentation":true,"enable-dotnet-instrumentation":true,"enable-go-instrumentation":true,"enable-python-instrumentation":true,"enable-nginx-instrumentation":false,"enable-nodejs-instrumentation":true,"enable-java-instrumentation":true,"zap-message-key":"message","zap-level-key":"level","zap-time-key":"timestamp","zap-level-format":"uppercase"}

However on pod creation, the init container is still injecting the public image:

Normal BackOff 41s (x43 over 10m) kubelet Back-off pulling image "ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.14.0-alpha"

Steps to Reproduce

1. Deployment of Operator via helm while enabling Go instrumentation specifying a container image for go instrumentation.
2. Deploy a collector.
3. Deploy an instrumentation CRD.
4. Deploy an application with the correct annotations for Go Injection

Expected Result

Expecting pods to be injected with the specified image.

Actual Result

The pods are still receiving the default public image regardless of the operator validating the correct image.

Kubernetes Version

v1.29.2

Operator version

v0.104.0

Collector version

v0.103.0

Environment information

Environment

AKS Deployment version v1.29.2

Log output

No response

Additional context

No response

[jaronoff97]

given you are deploying via helm, this is a known issue and I have an open PR to fix this. The change could potentially break users, so I'm still investigating the repercussions of this. For now, I would recommend setting the image manually in the Instrumentation CR like so:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: example
spec:
  exporter:
    endpoint: http://${OTEL_K8S_NODE_NAME}:4317
  go:
    image: <myimagehere>

[stephenhoran]

Ah I see. The instrumentation was given the annotation but the spec is not being patched. Redeploying the instrumentation corrected this as well. Given we are looking to deploy instrumentation across many namespaces I am glad this is reported and being worked on.