Description
In #5918 it was noted that there is a field definition collision between the Cloud Security Posture integration and the MongoDB datastream in the Network Packet Capture (NPC) integration. This is because both integrations place non-ECS fields in the root namespace.
This appears to be, in the case of NPC, due to Packetbeat predating extensive use of ECS for field definitions.
To address this and the problem more generally, both integrations should move these non-ECS fields into datastream-specific groups. This meta-issue covers the NPC side of this set of changes.
Proposed Approach
Stage 0. (immediately)
Duplicate type into the network.protocol ECS field before all other work.
Notify @brokensound77 when this is done so that detection rules can be updated.
Stage 1. (for 8.12)
Non-ECS fields in Network Packet Capture integration (NPC) datastreams will be placed in their own field group (an example of this is seen in a PR here) and generalised packetbeat-related fields will be placed in relevant ECS fields where appropriate (there is no example of this available yet).
Remapping to be performed in a conditional pipeline that user can opt into in the package's configuration UI. Initially the default will leave mappings in the old state, i.e. to not run the remapping pipeline. Clear documentation detailing the impacts of the choice must be included, including how it will impact on detection rules and so on.
Configuration to be per-datastream to allow progressive application of the proposed changes.
Searches and dashboards to be updated to use both the existing field locations and the the new ECS locations to allow users to continue to query/explore data from prior to the changeover.
Equivalent changes to be applied to the Packetbeat ingest pipelines and kibana assets.
- Completed network_traffic: add ECS compatibility pipelines and dashboards #8622, {,x-pack}/packetbeat: add support for pipeline module uploads beats#37291, packetbeat/docs: add documentation for ingest pipelines beats#37798
Stage 2 (for 8.13).
Mark old mappings as deprecated in the package documentation and in the configuration UI at the option to perform the remapping. This documentation to include instructions for users on how to make the change to ECS mappings, and a roadmap/timeline for the deprecation (this issue).
Provide a utility pipeline to convert the mappings of old — pre-change — documents to the new mapping via re-indexing for users who need to maintain historical data after the final deprecation and removal of the old mappings.
Stage 3 (stage 2. completion plus 6 months).
Change the default of the ECS-mapping pipeline config from off to on, making installations use the ECS mapping while leaving old installations in their existing state.
This stage will require coordination with Fleet (@nimarezainia) to ensure that UI notifications can be made to inform users of the change in behaviour before an upgrade is made.
Stage 4 (stage 3. completion plus 6 months).
Remove the original mappings and the option to choose which mapping to use. This change to be associated with a major version bump of the integration. Retain old search and dashboard behaviour until stage 5.
This stage will require coordination with Fleet (@nimarezainia) to ensure that UI notifications can be made to inform users of the change in behaviour before an upgrade is made.
BLOCKED: Requires elastic/kibana#187481
- Completed
Stage 5. (optional: no sooner than stage 4. completion plus 6 months)
Remove old search and dashboard behaviour.
This stage may not happen, but if it is decided that it will, will be preceded with clear notification and likely a further major version bump, again via UI notification.
- Completed/Dropped
/cc @siposea @andrewkroh @narph @epixa @nimarezainia @brokensound77