Skip to content

[enhancement] In esql validation, allow any order of metadata #4579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions detection_rules/rule.py
Original file line number Diff line number Diff line change
Expand Up @@ -914,8 +914,15 @@ def validates_esql_data(self, data, **kwargs):
query_lower = data['query'].lower()

# Combine both patterns using an OR operator and compile the regex
# The first part matches the metadata fields in the from clause by allowing one or multiple indices
# and any order of the metadata fields
# The second part matches the stats command with the by clause
combined_pattern = re.compile(
r'(from\s+\S+\s+metadata\s+_id,\s*_version,\s*_index)|(\bstats\b.*?\bby\b)', re.DOTALL
r'(from\s+(?:\S+\s*,\s*)*\S+\s+metadata\s+'
r'(_id,\s*_version,\s*_index|_id,\s*_index,\s*_version|_version,\s*_id,\s*_index|'
r'_version,\s*_index,\s*_id|_index,\s*_id,\s*_version|_index,\s*_version,\s*_id))'
r'|(\bstats\b.*?\bby\b)',
re.DOTALL
)

# Ensure that non-aggregate queries have metadata
Expand All @@ -927,7 +934,9 @@ def validates_esql_data(self, data, **kwargs):
)

# Enforce KEEP command for ESQL rules
if '| keep' not in query_lower:
# Match | followed by optional whitespace/newlines and then 'keep'
keep_pattern = re.compile(r'\|\s*keep\b', re.IGNORECASE | re.DOTALL)
if not keep_pattern.search(query_lower):
raise ValidationError(
f"Rule: {data['name']} does not contain a 'keep' command ->"
f" Add a 'keep' command to the query."
Expand Down