feat: add parser to allowlist for AWS global service IPs

[arthurzenika]

No description provided.

[arthurzenika]

These two expressions don't work yet. Looking for a way to check an IP against multiple ranges extracted from a JSON file.

[blotus]

File will read a file line by line, and return an array with the content of each line, the only way to reconstruct the JSON object I can think of would be to something like this, which is really not great :

let ip_json = Join(File("aws-ip-ranges.json"), '');
let jsonObject = JsonExtractSlice(ip_json, "prefixes");
...

[blotus]

Hello,

Unless I'm mistaken, this effectively allowlist all the AWS IPs ? (including EC2).

While allowing cloudfront is probably a good idea, allowing any EC2 IP seems kinda dangerous: even if AWS responds to abuse, some of their servers have malicious behaviours (AWS is in the top 5 AS in the community blocklist).

From a performance perspective, I'd expect this parser to be extremely slow: JsonExtract will reparse the JSON each time it is called, and there are about 15k ranges in the JSON file:

• Each log line will trigger parsing of a not-so-small JSON file
• In the worst case, the IP in the log line will be checked against 15k ranges, which will take some time.

Ideally, this file should be pre-processed and have a single range per line, which would allow to just use File and any without any parsing.
This is actually part of a larger subject: currently we have some scenarios that fetch datafiles from 3rd parties (eg, cloudflare ip ranges), I'd like to centralize everything in a repo we own (for example, https://github.com/crowdsecurity/sec-lists/) and have a CI job update everything automatically. This would allow for easy pre-processing if needed (and ideally, make sure crowdsec can use the same whitelists we use server-side for the consensus)

Additionally, this kind of resource-intensive match should be instead performed in a postoverflow: they will only be checked after a scenario was triggered, which should happen much less often.