[Logs+] Add API to create application logs integration during on-boarding

[weltenwort]

📓 Summary

Currently the log entries ingested after executing the "custom logs" on-boarding flow rely on the default logs-*-* templates shipped with Elasticsearch. That means the user can't customize the ingest pipeline or mapping for the specific dataset they're on-boarding. We want to provide an API that the workflow can use to create a integration for the newly on-boarded dataset.

• 🔬 based on research in: Prototype: API for creating an integration #158552

✔️ Acceptance criteria

• There is a fleet API that creates all the saved objects to bring a new integration into existence.
• The API respects the fleet plugin's conventions.
• The API handler re-uses the integration installation mechanism that already exists as much as possible.
• The API blocks until the installation is complete.
• The API reports the failure or success with appropriate status codes and response payloads.
• The API receives the following parameters:
  • integrationName: string
  • datasetType: 'logs' | 'metrics' | 'traces' | 'synthetics' | 'profiling'
  • datasetName: string
• The API creates the following assets:
  • package manifest with...
    • format_version: 2.9.0
    • name: integration name
    • title: capitalized integration name
    • description: "Collect logs for the dataset {dataset}."
    • version: 1.0.0
    • owner: {"github": "$currentKibanaUser"}
    • type: "integration"
    • conditions: {"kibana.version": ">=$currentKibanaVersion"}
  • datastream with...
    • datastream manifest
      • format_version: 2.8.1
      • dataset: dataset name
      • title: capitalized dataset name
      • type: dataset type
    • index template
      • derived from fields.yml
      • with basic fields
      • with agent fields
      • with reference to the common logs ILM policy
    • ingest pipeline
      • mimics the new default logs pipeline introduced in [Logs+] Default pipeline for logs data streams elasticsearch#95971, but for the specific dataset
• If the API call completes successfully the new integration is listed in the list of installed integrations.
• If the API call fails no orphaned assets are left behind.
• There are integration tests that cover this new API.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[weltenwort]

Open Questions

Is the manifest content accurate?

• User can't yet enter a title, a description, nor an icon during the on-boarding
• What are the semantics of the owner?
• Should we include a license?
• Should we set a type?
• Should we set conditions
• Should we set categories?

Verification

• The integrations UI warns about the integration not being signed. Is that acceptable or do we want Kibana to sign them?

DLM policy

• What should the DLM policy be?

[weltenwort]

@ruflin, your feedback would be welcome, especially about the manifest content

@yngrdyn, @ogupte, do you think this would work for the on-boarding workflow?

[ruflin]

Some quick notes:

• description: "Collect logs for the dataset {dataset} with Elastic Agent."
  • Can we leave out "with Elastic Agent" here?
• {"github": "elastic"}
  • I think ideally we should set a user here that matches the user logged in. If we can't, lets not make it elastic as it might be confusing when later a user exports, and elastic is indicated as the owner.
• manifest version: We need to find a way to keept his up-to-date with new versions of Kibana
• Verification: Yes, it is acceptable / expected, that is not signed. But we might change the UI for these slightly to make it clear, this is ok for your own integrations.
• DLM: Doesn't exist yet. The ILM policy by default should be the global logs ILM policy.

[weltenwort]

@ruflin thanks for the feedback!

Can we leave out "with Elastic Agent" here?

Sure, I just copied the package spec recommendation.

{"github": "elastic"}

The current user would work, I guess. When we add a publishing workflow at some point that should probably ask for the "real" value.

manifest version: We need to find a way to keep his up-to-date with new versions of Kibana

Isn't this the package spec version? Wouldn't we only have to update it when we use new features from the package spec?

The ILM policy by default should be the global logs ILM policy.

Do we copy it or refer to it?

[ruflin]

Isn't this the package spec version? Wouldn't we only have to update it when we use new features from the package spec?

Yes, it is the package spec versions. You are right that we only have to update it when we use a new feature. At the same time it would be nice, if newer versions of Kibana always create the "current" versions of package spec.

Do we copy it or refer to it?

Good question. So far in all integrations we refer to it. We should likely discuss if we should keep this. As soon as a user makes some modifications, we would copy. Eventually this is hopefully DLM and not ILM anymore which removes the problem.

[weltenwort]

Yes, it is the package spec versions. You are right that we only have to update it when we use a new feature. At the same time it would be nice, if newer versions of Kibana always create the "current" versions of package spec.

As far as I can tell, Kibana right now doesn't even know which package version it supports. @juliaElastic @jsoriano is there a mapping available to the code that represents the association between package spec and Kibana version?

[weltenwort]

Given that the packages have to specify the Kibana version, I've added a condition entry to the manifest ACs above. Does anyone see a problem with >=$currentKibanaVersion?

[ruflin]

Does anyone see a problem with >=$currentKibanaVersion

This LGTM. With serverless, the Kibana version will become irrelevant and the package version will be important for compatibility. So we will need both to be up-to-date. For now, lets do it manually. @jsoriano or @juliaElastic might have ideas on how to keep it in sync. My assumption would be that somewhere in the package manager in Kibana / Fleet, the supported version will be coded in eventually because of the above change.

[jsoriano]

That means the user can't customize the ingest pipeline or mapping for the specific dataset they're on-boarding. We want to provide an API that the workflow can use to create a integration for the newly on-boarded dataset.

I think that the custom logs input package (available since Kibana 8.8.0) already covers this.

When you create a policy with a custom dataset, Fleet already creates a logs-<dataset>@custom component template that can be used for customization.

It also creates a logs-<dataset>-<version> pipeline that references a logs-<dataset>@custom one:

It is true that this is not supported by current integration packages intended to be used as inputs. But I would rather consider this an area of improvement (I opened an issue about this #160775).

Is this new API still neccesary if input packages already provide this functionality?

[ruflin]

Is this new API still neccesary if input packages already provide this functionality?

Yes. The goal is to create a new integration. What currently happens with input packages is that the templates and pipelines are created, but I don't think belong then to an integration. In your example above, this would the foo integration. Also important, a Fleet policy is not part of our workflow.

[jsoriano]

Also important, a Fleet policy is not part of our workflow.

Ok, this is an important difference 👍

The use case is then for users that don't need (or want) to use neither Fleet nor pre-built integrations?

It may be good in any case to have alignment between how Fleet creates the dataset-specific templates and pipelines and how this new API will do it.

type: "integration"

Given that this is going to be more similar to an input package, should we use type: "input" instead? This will be probably simpler for an integration with a single data stream.

name: integration name
version: 1.0.0

What kind of integration names do we expect here? I wonder if there can be conflicts if this API is called for example with integrationName=apache and the user also tries to use the Apache package.

[ruflin]

The use case is then for users that don't need (or want) to use neither Fleet nor pre-built integrations?

Correct, they build their own.

Given that this is going to be more similar to an input package, should we use type: "input" instead? This will be probably simpler for an integration with a single data stream.

I think the opposite is true. Lets assume the nginx integration would not exist. The user would ship data to logs-nginx.access-default and then build templates / pipelines / dashboards on top of it. They just built an integration. Eventually, a user should be able to add more data streams to their integration.

What kind of integration names do we expect here? I wonder if there can be conflicts if this API is called for example with integrationName=apache and the user also tries to use the Apache package.

It is a problem we will hit. Eventually we will need something like a "namespace" prefix or something for the integrations. To get started, I would ignore it. A user just can't create an integration with a name that is already installed. Ideally also Fleet would not install an integration that already exists locally ...