Reference Sets/Lookups Proposal

[Res260]

I'd like to discuss the possibility of adding reference sets/lookups into the Sigma specification.

What are reference sets/lookups?

It's a list external to a query that you use into a query/detection.

SIEM	What it's called
QRadar	Reference set/map/table
Splunk	Lookups
Sentinel	Lookups/watchlists
Arcsight	Active lists
Logscale	lookup

Example of a rule that would use lookups

Here is a rule that would match when someone is added to a critical AD group:

logsource:
  product: Windows logs
detection:
  selection:
    EventID: 1234  # I dont know the event id lol
    TargetGroupName|lookup: 'critical_group_list'  # actual implementation must be discussed as well
  condition: selection

Why are they helpful and needed?

1. Lookups favor re-use of content accross rules. If we take the above example, we could have a rule when someone is removed from one of the same groups and re-use the list of critical groups.
2. Lookups keep a rule clean. Without lookups, a simple rule could get very unreadable if you need to lookup a list of 20+ items
3. Lookups work for a large amount of data. Lots of backend limit the maximum length of their query for performance and other reasons, which poses problems for large (50+) reference sets.

SIEM	Character limit per query
QRadar	<5,000
Splunk	None
Sentinel	10,000
Arcsight	idk
Logscale	66,000

Don't placeholders already solve this?

Here is the relevant part for the placeholders documentation:

Placeholders can be used to select a set of elements that can be expanded during conversion. Placeholders map a an identifier to a user defined value that can be set in config files for an automatic replacement during conversion runs. Placeholders are meaningful identifiers that users can easily expand themselves.

This is close from the use case I'm describing, but not quite it either:

• Sigma does not define the behavior of placeholders and uses the "this exercise is left to the reader" approach, which means 10 people reading this will likely create 10 different implementation of the same problem.
• This vagueness means no backend actually implement this feature and one must either create their own backend or use processing pipeline to try to make it work.
• Placeholders do not address point # 3, which is IMO the most important.

This goes against the SigmaHQ philosophy of "rules should be standalone and easily sharable"

This is true. However:

• Placeholders are worse for sharability and standaloneness yet they made their way into the Sigma specification
• In my opinion, the Sigma specification has broader use and is more important than the SigmaHQ project and concerns for the SigmaHQ project should not interfere with development of the Sigma specification, which is a major building block in detection-as-code implementation in many organizations, including mine. I love SigmaHQ and the work that's being done on this repo and frequently refer to it for reference, but I feel the rigidity of the contribution rules that come with it heavily influence the Sigma specification and I think, longer-term, this might become harmful to the specification.
• Reference sets/lookups can also easily be shared and they could even be bundled in the same file as the rule if needed.

Implementation proposal

Reference sets would be separate YAML files with a simple structure such as:

name: Critical AD Groups
type: string
entries:
  - Domain Admins
  - Enterprise Admins
  - Administrateurs du domaine
  - MyCustomAdminGroup1
  - ...

Rules could then reference them with something like a |lookup modifier:

logsource:
  product: Windows logs
detection:
  selection:
    EventID: 1234  # I dont know the event id lol
    TargetGroupName|lookup: 'Critical AD Groups'
  condition: selection

More things that need discussions:

• The discovery of reference sets needs to be discussed as well, maybe an extra argument to sigma convert
• Obviously sigma-cli should not be responsible for deployment of such reference sets, but sigma convert could take into account reference sets for output types like plugins or bundled applications like some backends support.
• Probably other things I didnt think about

Personal experience

We internally have implemented reference sets for our custom QRadar backend using placeholders. It works with a custom internal specification that looks like this: TargetGroupName|expand: '%refset:Critical AD Groups%'. This theoretically respects the Sigma specification, but there is a lot of custom stuff I needed to code to make it happen and think about it myself. I feel this is a common enough use case that we should standardize it into the Sigma specification, hence my proposal.

Let me know what you think!