Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ILM policy not created with data_stream_ilm_policy #1017

Open
2 tasks done
kobejn-jb opened this issue May 16, 2023 · 0 comments
Open
2 tasks done

ILM policy not created with data_stream_ilm_policy #1017

kobejn-jb opened this issue May 16, 2023 · 0 comments

Comments

@kobejn-jb
Copy link

(check apply)

  • read the contribution guideline
  • (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
    I'm using logging-operator, I've reported the issue on their slack

Problem

I'm using logging operator with flow: fluent-bit -> fluentd -> elasticsearch, my plan was to use fluentd config to manage ILM policies in ES, my first attempt was successful, here is config already processed by operator and prepared for fluentd:

  <match **>
    @type elasticsearch_data_stream
    @id clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch
    data_stream_ilm_name 5d-hot-15-warm-log-op
    data_stream_ilm_policy {
      "policy": {
        "phases": {
          "hot": {
            "min_age": "0ms",
            "actions": {
              "rollover": {
                "max_primary_shard_size": "20gb",
                "max_age": "1d"
              },
              "set_priority": {
                "priority": 100
              }
            }
          },
          "warm": {
            "min_age": "5d",
            "actions": {
              "set_priority": {
                "priority": 50
              }
            }
          },
          "delete": {
            "min_age": "15d",
            "actions": {
              "delete": {
                "delete_searchable_snapshot": true
              }
            }
          }
        }
      }
    }

    data_stream_name gke-lo-chesskid-staging
    exception_backup true
    fail_on_detecting_es_version_retry_exceed true
    fail_on_putting_template_retry_exceed true
    host es-int.tools-non-prod.chess-platform.com
    include_tag_key true
    password R9T7dMZOa0v8obkvkZl0ombk
    port 9200
    reconnect_on_error true
    reload_connections false
    reload_on_failure true
    scheme https
    ssl_verify true
    suppress_type_name true
    user elastic
    utc_index true
    verify_es_version_at_startup true
    <buffer tag,time>
      @type file
      chunk_limit_size 8M
      path /buffers/clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch.*.buffer
      retry_forever true
      timekey 10m
      timekey_wait 1m
      total_limit_size 5G
    </buffer>
  </match>

Defined ILM policy was created, but after I've added another "clusterOutput" in logging operator with another ILM policy it's simply not created (data stream and index template are created as expected and non existent ILM policy is set as lifecycle for indices in data stream)

Here are logs from fluentd configcheck pod:

fluentd -c /fluentd/etc/fluent.conf --dry-run
2023-05-16 13:18:31 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.1'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-azure-storage-append-blob' version '0.2.1'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.3.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-logdna' version '0.4.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.21'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2023-05-16 13:18:31 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-05-16 13:18:31 +0000 [info]: gem 'fluentd' version '1.14.6'
2023-05-16 13:18:31 +0000 [info]: starting fluentd-1.14.6 as dry run mode ruby="2.7.8"
2023-05-16 13:18:31 +0000 [info]: [clusterflow:logging:all-pods:0] DeDot will recurse nested hashes and arrays
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'host es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: es-int.tools-non-prod.chess-platform.com' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'host es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: es-int.tools-non-prod.chess-platform.com' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have timestamp placeholder for day('%d') for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] Need substitution: false
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'host_placeholder es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: es-int.tools-non-prod.chess-platform.com' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'host_placeholder es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: es-int.tools-non-prod.chess-platform.com' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'data_stream_name_placeholder gke-lo-chesskid-staging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: gke-lo-chesskid-staging' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch] 'data_stream_name_placeholder gke-lo-chesskid-staging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: gke-lo-chesskid-staging' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [info]: [clusterflow:logging:all-test:0] DeDot will recurse nested hashes and arrays
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'host es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: es-int.tools-non-prod.chess-platform.com' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'host es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: es-int.tools-non-prod.chess-platform.com' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have timestamp placeholder for day('%d') for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] Need substitution: false
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'host_placeholder es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: es-int.tools-non-prod.chess-platform.com' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'host_placeholder es-int.tools-non-prod.chess-platform.com' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: es-int.tools-non-prod.chess-platform.com' doesn't have tag placeholder
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'data_stream_name_placeholder gke-chesskid-test-staging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: gke-chesskid-test-staging' doesn't have timestamp placeholders for timekey 600
2023-05-16 13:18:31 +0000 [debug]: [clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2] 'data_stream_name_placeholder gke-chesskid-test-staging' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: gke-chesskid-test-staging' doesn't have tag placeholder
2023-05-16 13:18:32 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-05-16 13:18:32 +0000 [info]: using configuration file: <ROOT>
  <system>
    rpc_endpoint "127.0.0.1:24444"
    log_level debug
    workers 1
  </system>
  <source>
    @type forward
    @id main_forward
    bind "0.0.0.0"
    port 24240
  </source>
  <match **>
    @type label_router
    @id main
    metrics false
    <route>
      @label "@6457b2c7b5df52ec69b11e19290b49b6"
      metrics_labels {"id":"clusterflow:logging:all-pods"}
      <match>
        namespaces 
        negate false
      </match>
    </route>
    <route>
      @label "@9358c1108331342eb20a238e4c72ee0d"
      metrics_labels {"id":"clusterflow:logging:all-test"}
      <match>
        labels logs-storage-time:long
        negate true
      </match>
      <match>
        labels logs-storage-time:short
        negate true
      </match>
      <match>
        negate false
      </match>
    </route>
  </match>
  <label @6457b2c7b5df52ec69b11e19290b49b6>
    <filter **>
      @type dedot
      @id clusterflow:logging:all-pods:0
      de_dot_nested true
      de_dot_separator "_"
    </filter>
    <filter **>
      @type parser
      @id clusterflow:logging:all-pods:1
      key_name "message"
      remove_key_name_field true
      reserve_data true
      <parse>
        @type "json"
      </parse>
    </filter>
    <match **>
      @type elasticsearch_data_stream
      @id clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch
      data_stream_ilm_name "5d-hot-15-warm-log-op"
      data_stream_ilm_policy "{\"policy\":{\"phases\":{\"hot\":{\"min_age\":\"0ms\",\"actions\":{\"rollover\":{\"max_primary_shard_size\":\"20gb\",\"max_age\":\"1d\"},\"set_priority\":{\"priority\":100}}},\"warm\":{\"min_age\":\"5d\",\"actions\":{\"set_priority\":{\"priority\":50}}},\"delete\":{\"min_age\":\"15d\",\"actions\":{\"delete\":{\"delete_searchable_snapshot\":true}}}}}}"
      data_stream_name "gke-lo-chesskid-staging"
      exception_backup true
      fail_on_detecting_es_version_retry_exceed true
      fail_on_putting_template_retry_exceed true
      host "es-int.tools-non-prod.chess-platform.com"
      include_tag_key true
      password xxxxxx
      port 9200
      reconnect_on_error true
      reload_connections false
      reload_on_failure true
      scheme https
      ssl_verify true
      suppress_type_name true
      user "elastic"
      utc_index true
      verify_es_version_at_startup true
      <buffer tag,time>
        @type "file"
        chunk_limit_size 8M
        path "/buffers/clusterflow:logging:all-pods:clusteroutput:logging:elasticsearch.*.buffer"
        retry_forever true
        timekey 10m
        timekey_wait 1m
        total_limit_size 5G
      </buffer>
    </match>
  </label>
  <label @9358c1108331342eb20a238e4c72ee0d>
    <filter **>
      @type dedot
      @id clusterflow:logging:all-test:0
      de_dot_nested true
      de_dot_separator "_"
    </filter>
    <filter **>
      @type parser
      @id clusterflow:logging:all-test:1
      key_name "message"
      remove_key_name_field true
      reserve_data true
      <parse>
        @type "json"
      </parse>
    </filter>
    <match **>
      @type elasticsearch_data_stream
      @id clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2
      data_stream_ilm_name "5d-hot-15-warm-log-op-test"
      data_stream_ilm_policy "{\"policy\":{\"phases\":{\"hot\":{\"min_age\":\"0ms\",\"actions\":{\"rollover\":{\"max_primary_shard_size\":\"20gb\",\"max_age\":\"1d\"},\"set_priority\":{\"priority\":100}}},\"warm\":{\"min_age\":\"5d\",\"actions\":{\"set_priority\":{\"priority\":50}}},\"delete\":{\"min_age\":\"10d\",\"actions\":{\"delete\":{\"delete_searchable_snapshot\":true}}}}}}"
      data_stream_name "gke-chesskid-test-staging"
      exception_backup true
      fail_on_detecting_es_version_retry_exceed true
      fail_on_putting_template_retry_exceed true
      host "es-int.tools-non-prod.chess-platform.com"
      include_tag_key true
      password xxxxxx
      port 9200
      reconnect_on_error true
      reload_connections false
      reload_on_failure true
      scheme https
      ssl_verify true
      suppress_type_name true
      user "elastic"
      utc_index true
      verify_es_version_at_startup true
      <buffer tag,time>
        @type "file"
        chunk_limit_size 8M
        path "/buffers/clusterflow:logging:all-test:clusteroutput:logging:elasticsearch-2.*.buffer"
        retry_forever true
        timekey 10m
        timekey_wait 1m
        total_limit_size 5G
      </buffer>
    </match>
  </label>
  <label @ERROR>
    <match **>
      @type null
      @id main-fluentd-error
    </match>
  </label>
  <match **>
    @type null
    @id main-no-output
  </match>
</ROOT>
2023-05-16 13:18:32 +0000 [info]: finished dry run mode

Expected Behavior or What you need to ask

I'm expecting all ILM policies defined in the config would be created in ES but it's not happening, also I've deployed same logging stack on different cluster(common ES but separate k8s clusters) and even first ILM policy was not created there, I've tested with fluentd images v1.14 and v1.15. I've checked and I haven't done any changes in logging-operator config between the first defined policy that worked and latest one that didn't work. I'm not sure if this is a bug or am I doing something wrong

...

Using Fluentd and ES plugin versions

  • kubernetes v1.24.11-gke.1000 (GKE)
  • Fluentd v0.14.6 tested also on v0.15 container image
  • fluent-plugin-elasticsearch version '5.3.0'
  • ES version 8.4.3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kobejn-jb