[transform processor] How to get original metric resource within a chain of OTEL Collectors?

[gregbrowndev]

Hi all,

In my cluster, I've deployed a OTEL Collector "gateway", which contains OLTP receivers for push-based telemetry and centralises the processor and exporter logic.

I also have a small number of OTEL Collector "agents" deployed for specific services that have pull-based telemetry. These collectors simplify telemetry collection by sitting as sidecars to the main container and using the Prometheus receiver to scrape their metrics. These "agent" collectors export their telemetry to the "gateway" collector.

I want to use the Transform Processor in the gateway collector to add some common metric labels to all metrics using the resource attributes. Specifically, I want to ensure all metrics have the labels service_namespace and service_name (note: I can't add these labels in the instrumented service myself, but I can control the resource labels.) However, it seems that the upstream agent collector wraps the metric with its own resource object, or at least concatenates the resource service.name together.

So the metrics I see in Prometheus have the metric label: service_name: "otel-collector;history" when it should just say "history".

How can I read the Resource attributes from the leaf/original resource rather than this "otel-collector" resource?

OTEL Collector gateway config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  transform/metrics:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["service_namespace"], resource.attributes["service.namespace"])
          - set(attributes["service_name"], resource.attributes["service.name"])
          - set(attributes["service_instance_name"], resource.attributes["service.instance.name"])

  batch/metrics:
    timeout: 60s
    
  batch/traces:
    timeout: 1s
    send_batch_size: 50

exporters:
  prometheusremotewrite:
      endpoint: http://prometheus:9090/api/v1/write
      tls:
        insecure: true
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors:  
        - transform/metrics
        - batch/metrics
      exporters:
        - prometheusremotewrite
        - logging
    traces:
      receivers: [ otlp ]
      processors: [ batch/traces ]
      exporters:
        - otlp/jaeger

OTEL Collector agent config

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 10s
          static_configs:
            - targets: [0.0.0.0:8888]

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: "http://otel-collector-gateway:4317"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]