[Automatic Import] Safely access non-identifier fields in Painless if context

[ilyannn]

Release Note

Fixes how Automatic Import generates accesses for the field names that are not valid Painless identifiers.

Summary

Closes #205024

We add utility functions to access nested fields in Painless in a safe way and modify the existing ECS generation logic to use them.

This access happens using the object?.get("field") syntax for complex cases, while falling back to the familiar ctx.field for the cases where field is a valid Painless identifier and ctx is known to be non-nullable.

In the future this should be taken care of by the new $('a.b.c', defaultValue) accessor function (elastic/elasticsearch#101274). For now, it's not available:

{
  "error": {
...
    "type": "script_exception",
    "reason": "compile error",
    "processor_type": "set",
    "script": "$('cloud_provider', '') == 'azure'",
    "lang": "painless",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "invalid shortcut [$] for [field]; ensure [field] exists in this context"
    }
  },
  "status": 400
}

This takes care of the compile-time correctness of field accesses. Note that it is still possible for generated pipelines to fail in runtime on unexpected input, e.g. accessing a nested field a.b fails for the document of the form {"a": "string"}.

Testing

The two utility files we add are fully covered with unit tests:
.

Here's the generated package for logs containing only the @timestamp field:

• ai_202501022326__timestamp-1.0.0.zip

  - date:
      if: ctx.ai_202501022326__timestamp?.logs?.get("@timestamp") != null
      tag: date_processor_ai_202501022326__timestamp.logs.@timestamp
      field: ai_202501022326__timestamp.logs.@timestamp
      target_field: '@timestamp'
      formats:
        - yyyy-MM-dd'T'HH:mm:ss.SSSXXX
        - ISO8601

Checklist

• Unit or functional tests were updated or added to match the most common scenarios
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[haetamoudi]

Is the check for painless reserved keywords necessary? I get the regex check, but for example

Map test = new HashMap();
test.if = 'Reserved keyword';

return test.if;

do not throw any error for me, using the Painless Lab

[ilyannn]

That is a good point! I've taken this logic from the documentation that says:

Use an identifier as a named token to specify a ... field ...

and that a keyword cannot be an identifier, but perhaps the documentation is incomplete! I'll test whether this works specifically for the ingest pipelines and if yes I'll remove the check.

[ilyannn]

Yes when accessing the field all identifiers are fine:

ctx.if?.host // ok

though this is still understood as if statement:

if.something?.host // "unexpected token ['.'] was expecting one of ['(']."

[kibanamachine]

Starting backport for target branches: 8.16, 8.17, 8.x

https://github.com/elastic/kibana/actions/runs/12600012995

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2fdf792

Failed CI Steps

• FTR Configs #110

Test Failures

• [job] [logs] FTR Configs #110 / Spaces app space solution tour solution tour does not show the solution tour after deleting spaces and leave only the default

Metrics [docs]

✅ unchanged

cc @ilyannn

[kibanamachine]

💔 Some backports could not be created

Status	Branch	Result
❌	8.16	Backport failed because of merge conflicts You might need to backport the following PRs to 8.16: - [Infra][Hosts] Reset SearchBar refresh state to fully disable auto-refresh (#205416) - Upgrade cypress to 13.17.0 (#205437)
❌	8.17	Backport failed because of merge conflicts You might need to backport the following PRs to 8.17: - [Infra][Hosts] Reset SearchBar refresh state to fully disable auto-refresh (#205416) - Upgrade cypress to 13.17.0 (#205437)
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 205220

Questions ?

Please refer to the Backport tool documentation