Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update logs-config.md - structured_metadata #7026

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

Abuelodelanada
Copy link

According to what I was able to investigate, grafana-agent is not able to forward structured_metadata received from promtail (or another grafana-agent instance) to Loki.

https://grafana.slack.com/archives/C01050C3D8F/p1726151855977039

PR Description

Let's imagine we have this scenario:

promtail (or grafana-agent) --> grafana-agent --> loki

And Promtail has the following config in which we remove a label (filename) to add it as structured_metadata:

clients:
- tls_config:
    insecure_skip_verify: true
  url: http://agent-0.agent-endpoints.cos.svc.cluster.local:3500/loki/api/v1/push
positions:
  filename: /opt/promtail/positions.yaml
scrape_configs:
- job_name: system
  pipeline_stages:
  - drop:
      expression: .*file is a directory.*
  - structured_metadata:
      filename: filename
  - labeldrop:
    - filename
  static_configs:
  - labels:
      __path__: /bin/fake.log
      container: workload
      job: juju_cos_702ee831_flog
    targets:
    - localhost
server:
  grpc_listen_port: 9095
  http_listen_port: 9080

And these logs are received by a grafana-agent with a config like:

logs:
  configs:
  - clients:
    - tls_config:
        insecure_skip_verify: false
      url: http://192.168.1.240/cos-loki-0/loki/api/v1/push
    name: push_api_server
    scrape_configs:
    - job_name: loki
      loki_push_api:
        server:
          grpc_listen_port: 3600
          http_listen_port: 3500
      pipeline_stages:
      - drop:
          expression: .*file is a directory.*
      - structured_metadata:
          filename: filename
      - labeldrop:
        - filename
  positions_directory: /run/grafana-agent-positions

filename is not forwarded to loki as structured_metadata.

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

  • CHANGELOG.md updated
  • Documentation added
  • Tests updated
  • Config converters updated

According to what I was able to investigate, grafana-agent is not able to forward `structured_metadata` received from `promtail` (or another `grafana-agent` instance) to Loki.

https://grafana.slack.com/archives/C01050C3D8F/p1726151855977039
@CLAassistant
Copy link

CLAassistant commented Sep 16, 2024

CLA assistant check
All committers have signed the CLA.

@clayton-cornell clayton-cornell requested a review from a team September 17, 2024 18:47
docs/sources/static/configuration/logs-config.md Outdated Show resolved Hide resolved
docs/sources/static/configuration/logs-config.md Outdated Show resolved Hide resolved
@clayton-cornell
Copy link
Contributor

@mattdurham can you do a quick validity check on this one for Agent?

@mattdurham
Copy link
Collaborator

Looking at the code structured metadata should come through on both json and protobuf. We have a test for it here https://github.com/grafana/agent/blob/8908735a435029f915b7a7bfda1f8b9e7a3d9cd2/internal/component/loki/source/api/internal/lokipush/push_api_server_test.go

@clayton-cornell
Copy link
Contributor

@mattdurham does that mean the doc change proposed here is not accurate? IF someone having issues with structured metadata how can we resolve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/docs Docs Squad label across all Grafana Labs repos
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants