Breaking change to kubernetes.labels.app.*

[andrew-pickin-epi]

Describe the bug

Previously (v1.15) kubernetes.labels.app.* fields would have . replaced with app_.
Hence

kubernetes Label	record
kubernetes.labels.app.kubernetes.io/component	"kubernetes"=>{..., "labels"=>{"app_kubernetes_io/component"=>value}}
kubernetes.labels.app.kubernetes.io/instance	"kubernetes"=>{..., "labels"=>{"app_kubernetes_io/instance"=>value}}
kubernetes.labels.app.kubernetes.io/name	"kubernetes"=>{..., "labels"=>{"app_kubernetes_io/name"=>value}}

this was the default behaviour and (that I can tell) wasn't the result of some config, for example using the dedot plugin.
This changed at some point as of v1.16 resulting in

kubernetes Label	record
kubernetes.labels.app.kubernetes.io/component	"kubernetes"=>{..., "labels"=>{"app.kubernetes_io/component"=>value}}
kubernetes.labels.app.kubernetes.io/instance	"kubernetes"=>{..., "labels"=>{"app.kubernetes_io/instance"=>value}}
kubernetes.labels.app.kubernetes.io/name	"kubernetes"=>{..., "labels"=>{"app.kubernetes_io/name"=>value}}

Why this is important is many pods from third parties use kubernetes.labels.app as a text field in it's own right, now kubernetes.labels.app is a object field comprising many separate values. Elasticsearch now fails to accept these values because of this conflict.

To Reproduce

Run v1.16+ logging pods with both kubernetes.labels.app and kubernetes.labels.app.kubernetes.io/name labels.

Expected behavior

As described above as per v1.15 and prior.

Your Environment

v1.16-debian-elasticsearch8-amd64-2
same applies v1.17

Your Configuration

<match kubernetes.var.log.containers.**>
  @id rewrite_container_name
  @type rewrite_tag_filter
  <rule>
    key "$.kubernetes.container_name"
    pattern ^(.+)$
    tag "kubernetes.log.$1"
  </rule>
</match>
<filter kubernetes.log.**>
  @id parser_json
  @type parser
  key_name "log"
  reserve_data true
  remove_key_name_field true
  inject_key_prefix "log_"
  emit_invalid_record_to_error false
  <parse>
    @type "json"
  </parse>
</filter>
<match **> 
  @type copy
   <store>

    @type elasticsearch
    @id out_es7
    @log_level info
    include_tag_key true
    host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
    port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
    path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
    scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
...
  </buffer>
   </store>
</match>

Your Error Log

2024-09-18 15:45:29 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'failed to parse field [kubernetes.labels.app] of type [text] in document with id '_cPPBZIBw2MuyU6FXaXt'. Preview of field's value: '{kubernetes={io/component=controller}}''" location=nil tag="kubernetes.log..stdout" time=2024-09-18 15:45:23.589245851 +0000 record={"stream"=>"stdout", "logtag"=>"F", "log"=>"217.155.104.85,54.74.98.37 kibana-expt.epimorphics.net - [18/Sep/2024:15:45:23 +0000] \"GET /oauth2/auth HTTP/1.1\" 202 0 \"https://kibana-expt.epimorphics.net/app/discover\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36\" 3491 0.001 [monitoring-oauth2-proxy-4180] [] 192.168.79.20:4180 0 0.002 202 747ec619de8ae596356da98f27365cfe 1726674323.588 217.155.104.85", "docker"=>{"container_id"=>"bd7b51b011578e5cf0d581e6a0fc7826d530479a9df412a1f3ac0c03d47aee79"}, "kubernetes"=>{"container_name"=>"controller", "namespace_name"=>"default", "pod_name"=>"ingress-nginx-controller-f5b869567-88cxc", "container_image"=>"sha256:53c87e38a2091b8a026d977f962cbb533a89b0fa543738b40566fdb5f57cb758", "container_image_id"=>"registry.k8s.io/ingress-nginx/controller@sha256:8fd21d59428507671ce0fb47f818b1d859c92d2ad07bb7c947268d433030ba98", "pod_id"=>"b79d0bc4-81ad-4633-b28a-816582221f8c", "pod_ip"=>"192.168.15.164", "host"=>"ip-192-168-31-154.eu-west-1.compute.internal", "labels"=>{"app.kubernetes.io/component"=>"controller", "app.kubernetes.io/instance"=>"ingress-nginx", "app.kubernetes.io/managed-by"=>"Helm", "app.kubernetes.io/name"=>"ingress-nginx", "app.kubernetes.io/part-of"=>"ingress-nginx", "app.kubernetes.io/version"=>"1.9.3", "helm.sh/chart"=>"ingress-nginx-4.8.2", "pod-template-hash"=>"f5b869567", "s3_archive"=>"true"}, "master_url"=>"https://10.100.0.1:443/api", "namespace_id"=>"6ac07d57-033d-45c8-a570-f8a2451d22b4", "namespace_labels"=>{"kubernetes.io/metadata.name"=>"default"}}, "s3"=>"true", "log_format"=>".stdout", "@timestamp"=>"2024-09-18T15:45:23.589245851+00:00", "tag"=>"kubernetes.log..stdout"}

### Additional context

_No response_

[daipom]

Thanks for your report!

The specification of fluent-plugin-kubernetes_metadata_filter has changed since v3.0.0, and the do_dot feature has been removed.

Since the daemonset v1.16, that plugin version has been updated to v3.

So, we need to use fluent-plugin-dedot_filter.

It may be better that we should add some default configs to keep compatibility.

[daipom]

Related:

• Dedot implicitly removed from v1.16/debian-logzio #1442

[daipom]

We will tackle this in #1442.

[andrew-pickin-epi]

It might be of interest but I looked at this dedot filter in trying to correct this for my installation.
The dedot (the filter) has a flag (de_dot) which when set to false doesn't make the filter transparent, but always returns nothing, i.e. all records are lost.

I wasn't sure of the scope of the original filter, as it what fields it used to act on and why this had been removed, as in why it wasn't considered a good thing anymore.

Elasticsearch will take fields with dots. my problem was that I had kubernetes.labels.app (a string) and now was getting kubernetes.lables.app.kubernetes.io. (as map) etc, which upset Elasticsearch.

All the above caused me to end up re-writing dedot as https://github.com/epimorphics/fluent-plugin-flatten_filter to specifically target the labels.