{microsoft_defender_endpoint, m365_defender}: vulnerability data stream scaling problem #15521
Description
microsoft_defender_endpoint.vulnerability and m365_defender.vulnerability data streams consume huge memory while retrieving larger workloads.
The current approach retrieves all software vulnerabilities in user's environment by pulling data from 3 APIs before combining them, which isn't scalable.
APIs endpoints used inside the CEL program are:
/api/vulnerabilities/machinesVulnerabilities: Data retrieved is proportional to number of machines and softwares installed on them./api/machines: Data retrieved is proportional to number of machines./api/vulnerabilities: Data retrieved contains all known vulnerabilities.
Even for medium workloads (few thousand machines and/or softwares installed), the CEL input can hang without indexing the data.
Instead of pulling data from 3 APIs and combining them inside CEL program, a single API: Export software vulnerabilities assessment (via files) can fetch all software vulnerabilities from user's machines, and is the preferred approach:
This API solution enables pulling larger amounts of data faster and more reliably. Via-files is recommended for large organizations, with more than 100-K devices.
Moving to this new API there will be some fields missed from previous approach (event combining 3 APIs). But these missing fields wouldn't break the Security Vulnerabilities workflow. See #15521 (comment) for details on missing fields.