[Fleet] Packages with custom dataset use the fallback index templates

[jsoriano]

Kibana version:

Seen at least in 8.6, 8.7 and 8.8.

Describe the bug:

When the data stream of an integration package is configured to use a custom dataset, it may end up using the fallback index templates for logs and metrics, instead of the template installed with the package, missing important mappings or metadata.

This can cause issues like:

• Unexpected mappings for important fields (see [Log] Message field is keyword integrations#6566 (comment)).
• Data loss if the data cannot be ingested with the used mappings.
• Problems using Fleet features that rely on metadata. For example if the data stream doesn't include the package name in meta, it cannot be reused and may cause issues when migrating from integrations to input packages starting on 8.8.
• Apart of the mappings, no @custom pipeline is referenced, so no custom processing can be added.

This is confirmed at least with:

• The logs package (before 2.0), when the dataset is different to logs.logs.
• Prometheus package when the dataset is different to prometheus.collector.
• Windows Event Logs when the dataset is different to winlog.winlog.

Steps to reproduce:

1. Create an integration policy for an integration package that allows the use of custom datasets.
2. Use a custom dataset that is different to <package>.<data_stream>.
3. Ingest data.
4. Check how the created data stream is missing certain metadata such as the package name, and is using the logs or metrics built-in data streams.

Expected behavior:

The data stream uses the index template installed with the package. This is what happens with input packages or with integration packages that don't use custom datasets.

Screenshots (if relevant):

      GET _data_stream/metrics-prometheus*
      ...
      "_meta": {
        "managed": true,
        "description": "default metrics template installed by x-pack"
      },
      ...

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[MakoWish]

Any updates on this? I had a support case opened, and they pointed me here, but this looks to have gone stale.

[jlind23]

@MakoWish this isn't stale, it has been prioritised in on of our upcoming sprint but we have other priorities to deal with for now.

[hop-dev]

I think the solution here is to move to input packages where this problem is solved, I can see there is a prometheus_input package I wonder if doing the same is possible for winlog.winlog.

[juliaElastic]

Isn't it a bug in fleet that the package's index template is not used when custom dataset is used? If so, probably we should fix for all packages.

[hop-dev]

It is a bug, or just a bad feature I am not sure, dataset customisation has always been quite broken in integrations.

But Input packages were created to allow users to customize the destination of their data, if we want to allow integrations to customize dataset and have the index templates match it would be a new feature, one which I don't think we should invest time in when input packages exist.

[MakoWish]

The Custom Logs integration works perfectly fine with a custom dataset name, but the Custom Windows Event Logs one does not. Understanding one is reading files, and the other is reading Windows Event Logs, what's the difference in how those two manage a custom dataset name?

[hop-dev]

@MakoWish sorry I missed this. Custom Logs has been moved to an input type package, this means that fleet creates the index template dynamically for the given dataset. Whereas Windows event logs does not.

I have created an issue to move custom windows logs to an input package here elastic/integrations#7820

[jsoriano]

Even if we migrate these packages to input packages, this is still an issue for integration packages, right? Or should we disallow the use of custom datasets in integration packages?

[jsoriano]

Reopening as this is causing unexpected issues to users of integration packages that allow custom datasets.

I think we need to improve the behavior on these cases, or show a clear warning when creating policies that can drive to these situations.