[CrowdStrike Falcon] Flattened field needs to have ignore_above setting to avoid dropping events.

[leandrojmp]

Hello,

We are having some issues with some events being dropped because they have some immense field.

Flattened field [crowdstrike.event.Attributes] contains one immense field whose keyed encoding is longer than the allowed max length of 32766 bytes. Key length: 25, value length: 114376 for key starting with [old_group_assignment_rule]

Looking at the mappings for this field, the parameter ignore_above is not present.

"crowdstrike": {
        "properties": {
          "event": {
            "properties": {
              ( .... )
              "Attributes": {
                "type": "flattened"
              }
              ( .... )
            }
          }
        }
      }

Some keys in this flattened field can have a lot of values, so the ignore_above parameter needs to be added to at least avoid droppping events.

In the example the value for the key old_group_assignment_rule has 114 KB, so the ignore_above needs to be set to the maximum value allowed.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Created PR: #14127 for adding ignore_above option, but it is not getting reflected in the created index pattern.
Raised a bug inside elastic-package: elastic/kibana#223245

Alternate option to consider:

• Truncate very long fields inside Attributes field when they cross 32766 character length. Example:

  integrations/packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/default.yml

  Lines 3343 to 3345 in c8b50b2

  	} else if (src instanceof String && src.length() > 32766) {
  	return src.substring(0, 32700)+' (truncated)';
  	}

[leandrojmp]

@kcreddy I'm not sure, but wouldn't truncate lead to a mapping conflict?

If the flattened expects a json object, truncate will break the json object, which would lead to a mapping conflict dropping the document. (not tested this, so I'm not sure).

[kcreddy]

@kcreddy I'm not sure, but wouldn't truncate lead to a mapping conflict?

If the flattened expects a json object, truncate will break the json object, which would lead to a mapping conflict dropping the document. (not tested this, so I'm not sure).

@leandrojmp, the ignore_above option for a flattened field is applied on each underlying leaf field.
Similarly, we could iterate over Attributes and truncate each field's value inside it. We wont truncate Attributes field based on its size.
Do you think this workaround makes sense until a fix for elastic/kibana#223245 is implemented?

[kcreddy]

Blocked by elastic/kibana#223245