400 contains an unknown parameter [_type] status 400

[OliPou]

Problem

Hi team,

I'm facing an issue using elastic cloud on azure:

failed to flush the buffer. retry_times=6 next_retry_time=2024-12-23 13:39:52 +0000 chunk="629f016416b5885c2f4a23de7d18" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"somehost.francecentral.azure.elastic-cloud.com\", :port=>443, :scheme=>\"https\", :user=>\"my_user\", :password=>\"obfuscated\", :path=>\"\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}"

I installed the fluentd with the official helm chart and the following configuration:

fileConfigs:
  01_sources.conf: |-
    ## logs from podman
    <source>
      @type tail
      @id in_tail_container_logs
      @label @KUBERNETES
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>

    # expose metrics in prometheus format
    <source>
      @type prometheus
      bind 0.0.0.0
      port 24231
      metrics_path /metrics
    </source>

  02_filters.conf: |-
    <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>

      # <match kubernetes.var.log.containers.**_kube-system_**>
      #   @type null
      #   @id ignore_kube_system_logs
      # </match>

      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels false
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>

      <match **>
        @type relabel
        @label @DISPATCH
      </match>
    </label>

  03_dispatch.conf: |-
    <label @DISPATCH>
      <filter **>
        @type prometheus
        <metric>
          name fluentd_input_status_num_records_total
          type counter
          desc The total number of incoming records
          <labels>
            tag ${tag}
            hostname ${hostname}
          </labels>
        </metric>
      </filter>

      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>

  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type elasticsearch
        host "somehost.francecentral.azure.elastic-cloud.com"
        port 443
        scheme https
        ssl_verify true
        user my_user
        index_name my_index
        password my_password
        # Don't wait for elastic to start up.
        verify_es_version_at_startup false
        logstash_format true
        reload_connections false
        reconnect_on_error true
        reload_on_failure true
        suppress_type_name true
      </match>
    </label>

After reading issue on _type filed I discovered that I need to add the value suppress_type_name true
But even after I added this in my config files I still have the same problem.
...

Using Fluentd and ES plugin versions

• Debian GNU/Linux 12 \n \l
• Kubernetes (AKS)
• Fluentd fluentd 1.17.1
• ES plugin 3.x.y/2.x.y or 1.x.y
  abbrev (default: 0.1.1)
  addressable (2.8.7)
  base64 (0.2.0, default: 0.1.1)
  benchmark (default: 0.2.1)
  bigdecimal (default: 3.1.3)
  bundler (default: 2.4.19, 2.4.17)
  cgi (default: 0.3.6)
  concurrent-ruby (1.3.4)
  cool.io (1.9.0)
  csv (3.3.0, default: 3.2.6)
  date (default: 3.3.3)
  delegate (default: 0.3.0)
  did_you_mean (default: 1.6.3)
  digest (default: 3.1.1)
  domain_name (0.6.20240107)
  drb (2.2.1, default: 2.1.1)
  elastic-transport (8.3.5)
  elasticsearch (8.15.0)
  elasticsearch-api (8.15.0)
  english (default: 0.7.2)
  erb (default: 4.0.2)
  error_highlight (default: 0.5.1)
  etc (default: 1.4.2)
  excon (1.0.0)
  faraday (2.12.0)
  faraday-excon (2.3.0)
  faraday-net_http (3.3.0)
  fcntl (default: 1.0.2)
  ffi (1.17.0 x86_64-linux-gnu)
  ffi-compiler (1.3.2)
  fiddle (default: 1.1.1)
  fileutils (default: 1.7.0)
  find (default: 0.1.1)
  fluent-config-regexp-type (1.0.0)
  fluent-plugin-concat (2.5.0)
  fluent-plugin-dedot_filter (1.0.0)
  fluent-plugin-detect-exceptions (0.0.15)
  fluent-plugin-elasticsearch (5.3.0)
  fluent-plugin-grok-parser (2.6.2)
  fluent-plugin-json-in-json-2 (1.0.2)
  fluent-plugin-kubernetes_metadata_filter (3.5.1)
  fluent-plugin-multi-format-parser (1.0.0)
  fluent-plugin-parser-cri (0.1.1)
  fluent-plugin-prometheus (2.1.0)
  fluent-plugin-record-modifier (2.1.1)
  fluent-plugin-rewrite-tag-filter (2.4.0)
  fluent-plugin-systemd (1.1.0)
  fluentd (1.17.1)
  forwardable (default: 1.3.3)
  getoptlong (default: 0.2.0)
  http (5.2.0)
  http-accept (1.7.0)
  http-cookie (1.0.7)
  http-form_data (2.3.0)
  http_parser.rb (0.8.0)
  io-console (default: 0.6.0)
  io-nonblock (default: 0.2.0)
  io-wait (default: 0.3.0)
  ipaddr (default: 1.2.5)
  irb (default: 1.6.2)
  json (2.7.4, default: 2.6.3)
  jsonpath (1.1.5)
  kubeclient (4.12.0)
  llhttp-ffi (0.5.0)
  logger (1.6.1, default: 1.5.3)
  lru_redux (1.1.0)
  mime-types (3.6.0)
  mime-types-data (3.2024.1001)
  msgpack (1.7.3)
  multi_json (1.15.0)
  mutex_m (default: 0.1.2)
  net-http (default: 0.4.1)
  net-protocol (default: 0.2.1)
  netrc (0.11.0)
  nkf (default: 0.1.2)
  observer (default: 0.1.1)
  oj (3.15.1)
  open-uri (default: 0.3.0)
  open3 (default: 0.1.2)
  openssl (default: 3.1.0)
  optparse (default: 0.3.1)
  ostruct (0.6.0, default: 0.5.5)
  pathname (default: 0.2.1)
  pp (default: 0.4.0)
  prettyprint (default: 0.1.1)
  prometheus-client (4.2.3)
  pstore (default: 0.1.2)
  psych (default: 5.0.1)
  public_suffix (6.0.1)
  racc (default: 1.6.2)
  rake (13.2.1)
  rdoc (default: 6.5.1.1)
  readline (default: 0.0.3)
  readline-ext (default: 0.1.5)
  recursive-open-struct (1.3.1)
  reline (default: 0.3.2)
  resolv (default: 0.2.2)
  resolv-replace (default: 0.1.1)
  rest-client (2.1.0)
  rexml (3.2.9)
  rinda (default: 0.1.1)
  ruby2_keywords (default: 0.0.5)
  securerandom (default: 0.2.2)
  serverengine (2.4.0)
  set (default: 1.0.3)
  shellwords (default: 0.1.0)
  sigdump (0.2.5)
  singleton (default: 0.1.1)
  stringio (default: 3.0.4)
  strptime (0.2.5)
  strscan (3.1.0, default: 3.0.5)
  syntax_suggest (default: 1.1.0)
  syslog (default: 0.1.1)
  systemd-journal (2.0.0)
  tempfile (default: 0.1.3)
  time (default: 0.2.2)
  timeout (default: 0.3.1)
  tmpdir (default: 0.1.3)
  tsort (default: 0.1.1)
  tzinfo (2.0.6)
  tzinfo-data (1.2024.2)
  un (default: 0.2.1)
  uri (0.13.1, default: 0.12.2)
  weakref (default: 0.1.2)
  webrick (1.8.2)
  yajl-ruby (1.4.3)
  yaml (default: 0.2.1)
  zlib (default: 3.0.0)

[jammy-d]

Update verify_es_version_at_startup to true- it should work