How can I exclude pod logs from certain namespaces?

[cloudcafetech]

Using logging agent, is it possible to excludes pod logs from certain namespaces (default & kube-system)?

[mapno]

Using logging agent, is it possible to excludes pod logs from certain namespaces (default & kube-system)?

Yes. If you're using the kubernetes_sd_config to discover targets, you should be able to select only the resources you want to tail. For further manipulation, you can use relabel_configs to inspect and drop logs.

[cloudcafetech]

Any examples or pointer?

[rfratto]

(Hey there, I'm moving this to a discussion since this is a question)

As Mario said, relabel_configs is the best way to do this:

loki:
  configs:
  - name: default
    scrape_configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: default|kube-system
        action: drop 

This will drop any discovered target whose namespace is default or kube-system. The full list of avalable meta labels for Kubernetes can be found in the configuration reference.

[cloudcafetech]

Little confusing, where to put

              - job_name: kubernetes-pods-name
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - source_labels:
                      - __meta_kubernetes_pod_label_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-app
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                  - source_labels:
                      - __meta_kubernetes_pod_label_app
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-direct-controllers
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    separator: ""
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                  - action: drop
                    regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                  - source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-indirect-controller
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: .+
                    separator: ""
                    source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                  - action: keep
                    regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                  - action: replace
                    regex: ([0-9a-z-.]+)-[0-9a-f]{8,10}
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
              - job_name: kubernetes-pods-static
                kubernetes_sd_configs:
                  - role: pod
                pipeline_stages:
                  - docker: {}
                relabel_configs:
                  - action: drop
                    regex: ""
                    source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_label_component
                    target_label: __service__
                  - source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: drop
                    regex: ""
                    source_labels:
                      - __service__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: $1
                    separator: /
                    source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__

If I add on top below section what you mentioned its not excluding :(

              - job_name: kubernetes-pods
                kubernetes_sd_configs:
                  - role: pod
                relabel_configs:
                  - source_labels: [__meta_kubernetes_namespace]
                    regex: default|kube-system
                    action: drop

[rfratto]

You'll want to take the relabel config from my example:

                  - source_labels: [__meta_kubernetes_namespace]
                    regex: default|kube-system
                    action: drop

And paste it at the top of the relabel_configs for each of the jobs. Relabel configs are used to change labels and filter targets from SD, so by pasting my example job verbatim it left the other jobs alone.

[cloudcafetech]

@rfratto

It works 👍

Once again Thanks for your valuable time. 💯