Fleet: Custom integration creation / integration update fails on changing subobjects property on mapping

[flash1293]

When creating a custom integration (

kibana/x-pack/plugins/fleet/server/routes/epm/index.ts

Line 267 in e048b86

createCustomIntegrationHandler

) via /api/fleet/epm/custom_integrations for a datastream that already contains log datastreams created via the root logs template, the creation fails because the integration tries to set subobjects: false for all streams:

kibana/x-pack/plugins/fleet/server/services/epm/packages/custom_integrations/assets/dataset/utils.ts

Line 30 in 98aff21

subobjects: false,

As the existing datastreams don't have this set and it's not possible to change this setting after the fact, the creation fails with:

[2024-05-14T11:50:29.324+02:00][INFO ][plugins.fleet] Attempt to update the mappings for the logs-abcd.xyz-default (write_index_only)
[2024-05-14T11:50:29.333+02:00][ERROR][plugins.fleet] Mappings update for logs-abcd.xyz-default failed due to unexpected error: ResponseError: mapper_exception
Root causes:
mapper_exception: the [subobjects] parameter can't be updated for the object mapping [_doc]

The same happens when trying to upgrade an integration that has a data stream which didn't have subobjects defined in the old version but sets subobjects: false after the upgrade.

Steps to reproduce:

• Create a new dataset:

POST logs-myintegration.test-default/_doc
{
   "message": "abc"
}

• Go to /app/observabilityOnboarding/customLogs/?category=logs and create a new integration like thisL

clicking "Continue" fails:

I can think of the following fixes:

• Do not set subobjects: false for custom integrations for now
• Make it configurable
• Check whether there is matching data already and only set it if there is none
• Make it default in the central logs template (probably a dangerous change as it's so far reaching)

This is relevant for near-term plans around offering automated mapping issue mitigation flows.

cc @ruflin @achyutjhunjhunwala

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Hey @flash1293 thanks for the issue. The custom integrations code paths within Fleet were actually contributed by o11y (namely @Kerry350), so I'm a little wary of making changes here. To me, the Do not set subobjects: false for custom integrations for now option seems easiest to me, but I worry I'm not seeing the whole picture around custom integrations, so maybe Kerry can answer this better than me, or route us to someone who can.

Ref initial custom integrations scope: #159991

[flash1293]

Actually I added the subobjects: false recently - we want to make that the default anyway (see elastic/elasticsearch#106812), and I thought it would be a safe thing to do for the custom integrations - turns out it's not.

[ruflin]

As the existing datastreams don't have this set and it's not possible to change this setting after the fact, the creation fails with

Is just the update of the existing mapping failing or also if you set it on a template and then trigger the rollover?

[flash1293]

@ruflin

Is just the update of the existing mapping failing or also if you set it on a template and then trigger the rollover?

Only the existing mapping update for the current write index here:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

Line 1045 in ae997d9

await retryTransientEsErrors(

I tested the rollover case and it works as expected. So that would indeed be another option - to not attempt to update subobjects for the current write index, but instead just roll over.

[ruflin]

I tested the rollover case and it works as expected.

This is a common behaviour when upgrading Integration packages AFAIK. At first it tries to update the mapping and if we get an error, the template is updated and then a rollover is triggered as there are also other changes which require a rollover, like change to TSDS. The reason rollover is not always triggered is because this could lead to too many indices quickly.

[flash1293]

Ah, that's a good point @ruflin - we should definitely handle this case gracefully as it will also happen if an integration is switching over to subobjects: false. @kpollich what do you think about detecting this case and handling it gracefully?

[flash1293]

Updated the title to reflect that the scope is broader.

[kpollich]

what do you think about detecting this case and handling it gracefully?

Yes I think we should handle this via rollover as we do for standard integrations.

[kpollich]

Pulling this into the team's next sprint as a P1 bug. It should be relatively straightforward, and we should get some non-o11y folks into the custom integrations code regardless since we are codeowners on it 🙂

[flash1293]

@kpollich It's been a while this bug is open, is there anything to clear up about how to fix it? It's blocking some integration work.

[kpollich]

It's been a while this bug is open, is there anything to clear up about how to fix it? It's blocking some integration work.

I don't think there are any blockers here, it's just been tough to fit in with other priorities. Will try to make sure this is done as part of the current sprint. I'll bump the priority here.

For context - the Fleet team only oversaw the "custom integrations" work that was contributed to our API. We haven't actively maintained or contributed to these APIs to date, so this will be new ground for the team.

[criamico]

@flash1293 I tried to reproduce this bug locally following the steps in the description but I didn't get any error.

Followed steps:

POST logs-testintegration.test-default/_doc
{
   "message": "abc"
}

Is there any other steps that I'm missing? I tried twice but couldn't reproduce both times.

[flash1293]

@criamico In the second screenshot, the "dataset" name should be test, not testintegration.

In the data stream logs-testintegration.test-default you created, testintegration is the integration name, test is the dataset name and default is the namespace.

[criamico]

Thanks @flash1293, I have a PR with a tentative fix: #186991 Could you please take a look?