{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation

[kcreddy]

Proposed commit message

{microsoft_defender_endpoint, m365_defender}.vulnerability: New API implementation

Existing CEL program uses 3 API endpoints to fetch vulnerability
data. Although we fetch more fields using this approach, it doesn't 
scale well and hence is unusable even for few hundred machines.

This PR updates the vulnerability data stream with new 
SoftwareVulnerabilitiesExport API[1], which is recommended for 
larger workloads. While there are few data points missed in this 
new implementation[2], we maintain all the required fields for 
3rd party vulnerability workflow[3].

Other changes:
- Updates microsoft_defender_endpoint min stack version to "8.19.3"
  as the permissions for the transform were actually applied in 
  "8.19.3" version, and not in "8.19.2"[4].
- Add dataset filter to all visualisations of vulnerability dashboards.

[1]: https://learn.microsoft.com/en-us/defender-endpoint/api/get-assessment-software-vulnerabilities#2-export-software-vulnerabilities-assessment-via-files
[2]: https://github.com/elastic/integrations/issues/15521#issuecomment-3380969284
[3]: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide
[4]: https://github.com/elastic/elasticsearch/pull/132629

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Verified there are no prebuilt rules effected by this change.
• Update dashboards to remove references to old fields.
• Add video for upgrade flow.

How to test this PR locally

Pipeline tests and System tests pass.

--- Test results for package: m365_defender - START ---
╭───────────────┬───────────────┬───────────┬───────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE       │ DATA STREAM   │ TEST TYPE │ TEST NAME                                         │ RESULT │ TIME ELAPSED │
├───────────────┼───────────────┼───────────┼───────────────────────────────────────────────────┼────────┼──────────────┤
│ m365_defender │ vulnerability │ pipeline  │ (ingest pipeline warnings test-vulnerability.log) │ PASS   │ 480.561041ms │
│ m365_defender │ vulnerability │ pipeline  │ test-vulnerability.log                            │ PASS   │  87.309125ms │
╰───────────────┴───────────────┴───────────┴───────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: m365_defender - END   ---
Done

Related issues

• Closes {microsoft_defender_endpoint, m365_defender}: vulnerability data stream scaling problem #15521

Screenshots

Updated dashboards:

Upgrade Flow

mde-vulnerability-3.1-to-3.2-upgrade-flow.mp4

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package microsoft_defender_endpoint 👍(2) 💚(1) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
machine	3086.42	2392.34	-694.08 (-22.49%)	💔

To see the full report comment with /test benchmark fullreport

[brijesh-elastic]

Since there is a change in the transform schema, the fleet_transform_version and destination indices need to be updated.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: aa875d9

History

• 💚 Build #32584 succeeded 55b57a1
• 💔 Build #32583 failed 2f1492e
• 💔 Build #32578 failed c05ba2b
• 💚 Build #32564 succeeded f131c3e
• 💔 Build #32562 failed fe988d9
• 💔 Build #32561 failed 95dc8c8

cc @kcreddy

[elastic-vault-github-plugin-prod]

Package m365_defender - 5.0.0 containing this change is available at https://epr.elastic.co/package/m365_defender/5.0.0/

[elastic-vault-github-plugin-prod]

Package microsoft_defender_endpoint - 4.0.0 containing this change is available at https://epr.elastic.co/package/microsoft_defender_endpoint/4.0.0/