Remove deprecated data streams for 9.0

[narph]

Goal Statement

There are instances in Security Integrations where the third-party vendor has deprecated an API (some appear as a 'legacy'). These APIs are deprecated and not even accessible to customers.

These integrations are as follows:

• Microsoft 365 Defender - remove SIEM API
  
• Snyk - remove legacy API
  
• Carbon Black - remove httpjson input, which supports v6 API (Deprecated in Sept 2024)
  

Proposed Approach

The proposed approach would be to immediately ensure the configuration settings are marked as deprecated and then remove them entirely for 9.0.

Timeline

The proposed timeline is to remove these configurations for the 9.0 release of the Elastic stack.

Customer Impact

The deprecated APIs are no longer accessible to our customers.
This dashboard shows the customer using these integrations.

Integrations list

• Carbon Black Cloud
  • Remove Collect Carbon Black Cloud logs via API using HTTPJSON [Legacy]. It affects three data streams (alert, asset_vulnerability_summary, and audit).
  • Remove alert data stream. It has been replaced by alert_v7 relying on CEL.
  • Remove beta status for CEL input Collect Carbon Black Cloud logs via API using CEL [Beta].
  • Remove dashboard associated to alert data stream.
  • Update documentation.
• M365 Defender
  • Remove log data stream: M365D Incidents via SIEM API (Deprecated).
  • Update documentation.
• Snyk
  • Remove use of Legacy APIv1 and httpjson input (Collect data from Snyk API (Legacy))
  • That affects two data streams that are going to disappear, audit and vulnerabilities, that have been replaced by audit_logs and issues.
  • Update documentation.
• Remove f5 package.
  • According to its description: Deprecated. Use the F5 BIG-IP package instead.
• Remove Symantec EDR Cloud package.
  • It is marked as deprecated: Deprecated. Use the Symantec Endpoint Security package instead.

Open to discussion

• Anomali
  • The threatstream data stream was marked as deprecated a few months ago: [ti_anomali] Support the ThreatStream API #11309
  • From PR's description: The two data sources are roughly equivalent, although there are significant differences in the data format and some other details.
  • @chrisberkhout do you think is a good opportunity to completely remove the deprecated data stream? or still too soon?
  • Update: waiting for feedback from Anomali.
• Cisco Duo
  • We cannot remove the use of httpjson as some data streams still use it to access the v1 API because there are no v2 endpoints for these data streams yet.
  • Remove the telephony data stream, as we already have telephony_v2. Would it be a problem to rename telephony_v2 to telephony at this point? It would imply renaming fields, dataset name and dashboards.
  • Remove the httpjson input for the Auth data stream, as it was already migrated to CEL.
  • Update documentation.
  • Update: Telephony v2 is still under Tech Preview.
• Jamf Protect
  • Remove telemetry_legacy data stream? According to [Jamf Protect] - Adding support for new Telemetry Data Stream #10152, it says that the Telemetry feature will migrate over to a new data model but doesn't specify if the new changes are available yet.
  • Update: Do not remove the telemetry_legacy data stream for now.
• O365
  • Remove use of o365audit input for the Audit data stream: Collect Office 365 audit logs - Deprecated. Please disable this and use the CEL input instead.
  • Update documentation.
  • Question: move general settings to package level (e.g. base url, tenant ID, client credentials, etc.)?
  • There is a WIP that could be in conflict with these changes: https://github.com/elastic/enhancements/issues/23090. @ShourieG, is this audit data stream going to be deprecated after the planned changes?

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

The release of these changes should coincide with the timing of the 9.0 Elastic stack release. We should prepare the changes before then so that they are ready, but we hold off on the merge until 9.0 ships.

[narph]

[narph]

@jsoriano , what are the consequences of removing data streams in serverless/non-serverless?
Do we support a smooth upgrade workflow where users are redirected to the page to update their configuration? (will need to test this scenarios)

[jamiehynds]

• F5: Agree. Old RSA2ELK package where we have a much better alternative for.
• Symantec: Agree. Customers should be using the Symantec Endpoint Security integration instead.
• Anomali: TBD. Pinged Anomali to ask about adoption of the Anomali Integrator for SIEM integrations. I think we may need to revert the deprecation, and continue to support both the Integrator SDK and API, but will update once we hear back from Anomali.
• Cisco Duo: Telephony v2 is still under Tech Preview. Suggest we continue to support both V1 and V2, until V1 is deprecated and V2 is GA.
• Jamf Protect: input from @txhaflaire needed.
• O365: Not opposed to deprecation of the httpjson based data stream, but need to be very clear in how we message to users and what the upgrade flow looks like. Can't afford to risk breaking users ingestion of O365 events, due to lack of warning that they need to reconfigure their integration post-upgrade.

[txhaflaire]

@jamiehynds Related to Jamf Protect - there are customers out there that are using the V1 feature still as they have not yet migrated. I would recommend that we keep the V1 (telemetry_legacy) in for a bit still.

[jsoriano]

The release of these changes should coincide with the timing of the 9.0 Elastic stack release. We should prepare the changes before then so that they are ready, but we hold off on the merge until 9.0 ships.

@andrewkroh what is the plan to apply these changes? Is it to remove the data stream and set kibana.version to ^9.0.0 at the same time?
In this case it wouldn't be needed to align the release so much, as the change would not be present for older versions.

what are the consequences of removing data streams in serverless/non-serverless?

We have three ways to control availability of packages:

• Kibana version constraints.
• Spec version constraints, format_version in package manifests.
• Stack capabilities (not relevant here)

Serverless only uses spec version constraints, it ignores kibana version constraints.

I guess that when we talk about removing data streams in 9.0, we refer to removing data streams in packages while we change their kibana version constraints to include 9.x versions.

Removing the data streams this way will make the data streams to silently disappear for users of serverless.

Do we support a smooth upgrade workflow where users are redirected to the page to update their configuration? (will need to test this scenarios)

Workflow is "smooth" now. If a data stream is removed now, the user can complete the upgrade without seeing any warning, and the data stream is effectively removed from their config. This is probably fine for cases where the data stream is broken now, but it may be too smooth for the cases where the users are using the removed data stream and should start using a different one. Users won't receive any explicit notification about their configs being removed.

Expand for details on current behavior

Tried now to remove a policy template from the apache package, and the main problem may be that no warning is reported to users. In some cases this may be good, because it is mostly transparent, but if the removed data stream was working for them, it may result in missing data.

During the upgrade, the only moment you can see that there is a breaking change, is if you click to see the changelog, it is shown like this:

I marked the checkbox to upgrade policies, and then I clicked to upgrade, and the upgrade succeeded, without any warning. But the policy was not automatically upgraded. Not sure if there is something in Fleet checking that things were being removed.
@kpollich do you know if we have something detecting data stream removals during upgrades?

If then I go to upgrade the policy, I see that it is ready to upgrade, and everything is good, but the removed httpjson data stream doesn't appear there. If I save there, the data stream is effectively removed from the config, and the user doesn't receive any warning.

We have some proposals to try to improve the experience on this case:

• Add more information in packages about deprecated features, with possibility of migration paths: [Discuss] Support providing hints for breaking changes package-spec#817
• Show breaking changes more prominently to users when upgrading: [Fleet] Detect breaking changes in integration change logs and flag them to users during integration upgrades kibana#187481

I think we should at least to warn users when upgrading to a version with breaking changes.
Adding hints with migration paths could even help to provide automatic migrations for deprecated data streams that have a replacement.

As it is late to make changes in Fleet for 9.0, I see these options:

• We accept the current situation, remove the data streams in alignment with 9.0 and do nothing now in Fleet. Then users won't be notified about these removals unless they check changelogs or other release notes.
• We do something like [Fleet] Detect breaking changes in integration change logs and flag them to users during integration upgrades kibana#187481 asap, before 9.0 is released and data streams are removed, so at least users of serverless see the warnings on breaking changes.
• We don't remove the data streams that have a replacement yet, and plan an automated migration path for 9.x based on [Discuss] Support providing hints for breaking changes package-spec#817, so the change is not so breaking for any user of the removed data streams.

ccing @nimarezainia @kpollich about the feasibility of planning these changes in the Spec and Fleet.

[jamiehynds]

@jamiehynds Related to Jamf Protect - there are customers out there that are using the V1 feature still as they have not yet migrated. I would recommend that we keep the V1 (telemetry_legacy) in for a bit still.

Sounds good, thanks for the context @txhaflaire.

@chemamartinez @chemamartinez can you ensure that the Jamf Protect remains as-is, and we don't remove the v1 telemetry until further instruction from Jamf.

[chrisberkhout]

@chemamartinez asked:

@chrisberkhout do you think is a good opportunity to completely remove the deprecated data stream? or still too soon?

@jamiehynds said:

Anomali: TBD. Pinged Anomali to ask about adoption of the Anomali Integrator for SIEM integrations. I think we may need to revert the deprecation, and continue to support both the Integrator SDK and API, but will update once we hear back from Anomali.

It makes sense to coordinate with Anomali to ensure we don't remove the legacy data stream while the Elastic Extension software is still being offered to users. In general the new data stream is better and easier to set up, so I look forward to removing the legacy one if Anomali gives the go-ahead.

[narph]

@jamiehynds , @andrewkroh , we've deprecated the o365audit input 11 months ago, if we decide to remove the deprecated o365 datastream, should we consider removing the input as well (it's the only integration that is being used)?

[chemamartinez]

@narph @jsoriano @jamiehynds @andrewkroh

What would be the best strategy here to remove data streams from packages or apply any other breaking change? based on recent comments

Is it to remove the data stream and set kibana.version to ^9.0.0 at the same time?
In this case it wouldn't be needed to align the release so much, as the change would not be present for older versions.

I guess that when we talk about removing data streams in 9.0, we refer to removing data streams in packages while we change their kibana version constraints to include 9.x versions.

The problem I see with forcing packages where we delete data streams to support only 9.0 or higher, is that we will force users with these integrations to update the entire stack when we add enhancements or fixes on top of it, or we will have to continually backport the fixes to be included in these integrations.

Another option is to continue to support 8.x and 9.0 in these integrations despite introducing a breaking change, which would make the breaking change not aligned with the 9.0 release.

[jsoriano]

The problem I see with forcing packages where we delete data streams to support only 9.0 or higher, is that we will force users with these integrations to update the entire stack when we add enhancements or fixes on top of it, or we will have to continually backport the fixes to be included in these integrations.

Exactly, if the data streams are removed, and the new versions support 9.0 only, the change will be aligned with 9.0, but you will need to backport fixes.

But even in this case, the data streams removal will affect serverless, where no version constraint applies (serverless is also "versionless" 🙂)

What would be the best strategy here to remove data streams from packages or apply any other breaking change?

Not sure if we have telemetry about usage of data streams, but if we have, I think I would avoid removing any data stream that is being used and is known to work properly. We may affect users that may not know that they are affected till too late.

So, for any deprecated data stream or package:

• If it is known to be broken (because APIs have changed or something like that), I think it can be removed, no need to align with 9.0.
• If it is not known to be used, we can silently remove it, adding the breaking change to the changelog.
• If it works, and it is used, I think we should not remove it. If it has a replacement, I think we should wait for something like [Discuss] Support providing hints for breaking changes package-spec#817, so we can try to make the change not so breaking for the user.

It looks to me that aligning the removal with 9.0 gives little benefits, taking into account that serverless exists, and that we would like to support a broad range of stack versions with the same packages.

[jamiehynds]

Regarding the integrations 'above the line', namely Carbon Black Cloud, Snyk, M365 Defender, F5 and Symantec EDR. I think we're fine to go proceed with removal as the API's have been deprecated by each vendor, or we have newer packages to move to (F5/Symantec). Silently removing it and adding the breaking change to the changelog may be the easiest approach.

Can we leave the integrations below the line untouched - in most cases the API's are still valid and work, and not a strong case for deprecation.

[colleenmcginnis]

👋 @jamiehynds do we need to continue publishing docs pages for the deprecated integrations after the 9.0.0 release? Our current docs automation isn't set up to do this since we use https://epr.elastic.co/search?kibana.version={latest_kibana_version} to get the relevant integrations and https://epr.elastic.co/search?kibana.version=9.0.0 doesn't return any results for the deprecated integrations.

cc @alaudazzi

[jamiehynds]

Hey @colleenmcginnis - we're fine to remove the docs, but just need to be careful to remove the right docs, as it could be easy to mix them up As an example, we remove these F5 docs but keep these F5 docs.