Extend remote_write config with X-Scope-OrgID

[till]

I am looking for a mechanism to extend the remote_write configuration of the agent and add a custom header (for X-Scope-OrgID) based on source label. Similar to promtail where I can configure it to split logs based on labels (into different tenants) and send the header along to Loki. Couldn't find anything in the documentation, but maybe someone has an idea.

[rfratto]

Hi! The remote_write for metrics and traces support adding custom headers.

Metrics example:

metrics:
  configs:
  - name: example
    remote_write:
    - url: http://example.com
      headers:
        X-Scope-OrgID: example-org-id

Traces example:

traces:
  configs:
  - name: example
    remote_write:
    - endpoint: example.com:443
      headers:
        X-Scope-OrgID: example-org-id

[till]

Thanks for responding. :)

Yeah, we are using that already, but I found nothing that let's me set the header based on metric/trace.

That's what I am looking for. Like a rewrite of the header based on a label value so I can split data based on source in to different tenants.

Consider different namespaces that run different applications and I want to have data for these in different tenants.

For Loki we use promtail, to do the rewrite based on pod labels.

[rfratto]

Ah, sorry, I missed the "based on source label" part.

Right, that's unfortunately not something supported today, and would have to be changed upstream by Prometheus for us to have an equivalent functionality.

A gnarly workaround would be to use write_relabel_configs and write to a proxy which unpacks the label into a header, but that's not a small amount of work.

[till]

Yeah, no worries. It's been a long week. Just doing research if I couldn't find something.

I found this project for cortex:
https://github.com/blind-oracle/cortex-tenant

Will probably explore, was hoping I missed something and I wouldn't have to run more things.

[mvadu]

Hi! The remote_write for metrics and traces support adding custom headers.

Metrics example:

metrics:
  configs:
  - name: example
    remote_write:
    - url: http://example.com
      headers:
        X-Scope-OrgID: example-org-id

Traces example:

traces:
  configs:
  - name: example
    remote_write:
    - endpoint: example.com:443
      headers:
        X-Scope-OrgID: example-org-id

@rfratto you called out only metrics and traces, is it possible to add a custom header for loki as well (i.e. does clients config allow us to specify the cusom header?)

[rfratto]

Custom headers for logs doesn't look possible at the moment; it'd have to be an upstream PR / request to Promtail since we just expose its config directly.