Restart OTEL Collector on change to imported ConfigMap

[lsolovey]

We configure OTEL Collector to read specific sub-sets of configuration from the separate "imported" file, using ${file:...} syntax (as documented here).

The imported file is mounted as a volume from a separate ConfigMap. See the definition of OTEL Collector CRD below, simplified for clarity.

The question is: how do we force OTEL Collector to restart (or reload the config) upon any change to the imported ConfigMap?

Manual restart of OTEL Collector (kubectl rollout restart) doesn't work, as explained in this issue: #1300

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otelcollector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
    processors:
      # Tail sampling configuration is externalized in the separate ConfigMap
      tail_sampling: ${file:config/tail-sampling.yaml}
    exporters:
      logging: {}
    service:
      pipelines:
        traces:
          receivers:
          - otlp
          processors:
          - tail_sampling
          exporters:
          - logging
  volumeMounts:
  # Mount ConfigMap which contains tail-sampling.yaml
  - mountPath: /config
    name: tail-sampling-configmap
  volumes:
  # Mount ConfigMap which contains tail-sampling.yaml
  - configMap:
      name: tail-sampling
      optional: true
    name: tail-sampling-configmap

FYI we use "imported" files because different parts of OTEL Collector are configured by different teams with different release cycles. Hence, we want to split such parts of configuration into separate imported ConfigMaps, to allow teams working independently.

[pavolloffay]

See #1300 (comment).

[jaronoff97]

I think #1989 may also assist here

[swiatekm]

Watching external ConfigMaps for changes is outside the operator's responsibilities in my opinion. If you want this to happen, you should use one of the standard mechanisms - either add a Pod Annotation with the hash of your ConfigMaps' content, or have the ConfigMap name itself contain the hash (kustomize has built-in support for this).

If I may ask, why are you managing the collector configuration this way? This problem would be easiest to solve by making the operator manage it directly, and we are considering adding more elaborate configuration management features.

[lsolovey]

If I may ask, why are you managing the collector configuration this way?

I tried to explain the reason in the original post:

FYI we use "imported" files because different parts of OTEL Collector are configured by different teams with different release cycles. Hence, we want to split such parts of configuration into separate imported ConfigMaps, to allow teams working independently.

In other words, current OTEL Operator configuration is somewhat monolithic. It doesn't work well for large organizations with different departments managing only particular aspects of configuration.

This problem would be easiest to solve by making the operator manage it directly, and we are considering adding more elaborate configuration management features.

@swiatekm-sumo Could you please elaborate, what do you mean by " making the operator manage it directly". I'm not sure I understood your suggestion. Maybe I'm missing something. Thanks!

[swiatekm]

@swiatekm-sumo Could you please elaborate, what do you mean by " making the operator manage it directly". I'm not sure I understood your suggestion. Maybe I'm missing something. Thanks!

You can read my initial thoughts on this problem here: #1906. We're still discussing how to actually approach this problem, so it would help if you could comment in that issue and explain your use case in more detail.

For your specific use case as I understand it, we could add new Custom Resources for managing configuration. Then each of your teams could manage their configuration via their own CR, and you'd just reference those in your OpenTelemetryCollector CR. Somewhat similar to how ServiceMonitors work with prometheus-operator. Then you'd automatically get collector restarts on any change to these CRs.

[lsolovey]

Thanks @swiatekm-sumo, #1906 is exactly what I'm thinking about (in the long-term). I'll post our use-case there.

For now, I'm experimenting with podAnnotations to add our ConfigMap hash, as you, @pavolloffay and @jaronoff97 suggested.

[jaronoff97]

great, thank you! Closing this issue :)