Skip to content

[Rule Tunings] Reduce Usage of Flattened Fields in AWS Rules #4892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

[Rule Tunings] Reduce Usage of Flattened Fields in AWS Rules #4892

wants to merge 13 commits into from

Conversation

imays11
Copy link
Contributor

@imays11 imays11 commented Jul 10, 2025 •
edited
Loading

Pull Request

Issue link(s):

Summary - What I changed

This PR is in part a response to the following issues: elastic/integrations#13500 elastic/integrations#14236 regarding the future of flattened fields in AWS, which we use as an essential part of our ruleset. However, this is also in response to the ongoing ruleset audit. Some of the flattened fields used in queries are not truly necessary for the alert to trigger or can be replaced by a different field. Those changes have been made here and our non_ecs file has been edited to remove the unnecessary fields. Additionally, flattened fields have been removed from highlighted fields, and from investigation guides. The fields that remain in non_ecs file are necessary for rules and there are no existing alternatives for this data.

Note about "Insecure AWS EC2 VPC Security Group Ingress Rule Added"

  • this rule was not triggering as expected because the flattened.request_parameter field was not properly mapped in the rule. It's been updated in the rule and in the ecs_file. Below is the screenshot of working query.
Screenshot 2025-07-10 at 11 53 00 AM

Note about "AWS EC2 Security Group Configuration Change"

  • this rule was missing a related event.action AuthorizeSecurityGroupIngress . ModifyInstanceAttribute is an API call that allows for modifying various aspects of an EC2 instance, left this broad it was capturing behaviors that are not associated with security group modification. a flattened.request_parameters field was added to distinguish this behavior and was also added to ecs_file. There is no alternative data source for this information. Below is a screenshot of the new working query
Screenshot 2025-07-10 at 12 40 18 PM

How To Test

Each of these rules has an accompanying script here that can be used to trigger the rule for testing. Each rule has been tested to ensure it's triggering as expected.

@imays11
[Rule Tunings] Reduce Usage of Flattened Fields in AWS Rules
8752565 
This PR is in part a response to the following issues regarding the future of flattened fields in AWS, which we use as an essential part of our ruleset. However, this is also in response to the ongoing ruleset audit. Some of the flattened fields used are not truly necessary for the alert to trigger or can be replaced by a different field. Those changes have been made here and our non_ecs file has been edited to remove the unnecessary fields. Additionally, flattened fields have been removed from highlighted fields, and from investigation guides.
@imays11 imays11 self-assigned this Jul 10, 2025
@imays11
Update non-ecs-schema.json
969e1cc 
add necessary field for ModifyInstanceAttribute action
@imays11
Update persistence_ec2_security_group_configuration_change_detection.…
bff2499 
…toml

added missing event.action AuthorizeSecurityGroupIngress, narrowed scope for ModifyInstanceAttribute action by adding a necessary flattened_field
@imays11
Update privilege_escalation_iam_update_assume_role_policy.toml
949577d 
updating min_stack to account of target.entity.id field
@imays11 imays11 added Integration: AWS AWS related rules Rule: Tuning tweaking or tuning an existing rule Team: TRADE Domain: Cloud labels Jul 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Rule: Tuning - Guidelines

These guidelines serve as a reminder set of considerations when tuning an existing rule.

Documentation and Context

  • Detailed description of the suggested changes.
  • Provide example JSON data or screenshots.
  • Provide evidence of reducing benign events mistakenly identified as threats (False Positives).
  • Provide evidence of enhancing detection of true threats that were previously missed (False Negatives).
  • Provide evidence of optimizing resource consumption and execution time of detection rules (Performance).
  • Provide evidence of specific environment factors influencing customized rule tuning (Contextual Tuning).
  • Provide evidence of improvements made by modifying sensitivity by changing alert triggering thresholds (Threshold Adjustments).
  • Provide evidence of refining rules to better detect deviations from typical behavior (Behavioral Tuning).
  • Provide evidence of improvements of adjusting rules based on time-based patterns (Temporal Tuning).
  • Provide reasoning of adjusting priority or severity levels of alerts (Severity Tuning).
  • Provide evidence of improving quality integrity of our data used by detection rules (Data Quality).
  • Ensure the tuning includes necessary updates to the release documentation and versioning.

Rule Metadata Checks

  • updated_date matches the date of tuning PR merged.
  • min_stack_version should support the widest stack versions.
  • name and description should be descriptive and not include typos.
  • query should be inclusive, not overly exclusive. Review to ensure the original intent of the rule is maintained.

Testing and Validation

  • Validate that the tuned rule's performance is satisfactory and does not negatively impact the stack.
  • Ensure that the tuned rule has a low false positive rate.

@imays11 imays11 marked this pull request as ready for review July 10, 2025 17:29
@imays11 imays11 added patch and removed patch labels Jul 10, 2025
Aegrah
Aegrah approved these changes Jul 14, 2025
View reviewed changes
Copy link
Contributor

@Aegrah Aegrah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just some nits!

rules/integrations/aws/impact_s3_unusual_object_encryption_with_sse_c.toml
extort the bucket owner for the decryption key. This is a [New
Terms](https://www.elastic.co/guide/en/security/current/rules-ui-create.html#create-new-terms-rule) rule that flags when
this behavior is observed for the first time in the last 14 days by the user ARN and target bucket name.
Identifies when AWS S3 objects stored in a bucket are encrypted using Server-Side Encryption with Customer-Provided Keys (SSE-C). Adversaries with compromised AWS credentials can encrypt objects in an S3 bucket using their own encryption keys, rendering the objects unreadable or recoverable without the key. This can be used as a form of ransomware to extort the bucket owner for the decryption key. This is a New Terms rule that flags when this behavior is observed for the first time user and target bucket name.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the target bucket the tls.client.server_name field in the new_terms set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Aegrah Yes it is

terrancedejesus
terrancedejesus reviewed Jul 14, 2025
View reviewed changes
rules/integrations/aws/privilege_escalation_iam_update_assume_role_policy.toml
@@ -125,7 +126,7 @@ reference = "https://attack.mitre.org/tactics/TA0004/"

[rule.new_terms]
field = "new_terms_fields"
value = ["cloud.account.id", "user.name", "aws.cloudtrail.flattened.request_parameters.roleName"]
value = ["cloud.account.id", "user.name", "target.entity.id"]
Copy link
Contributor

@terrancedejesus terrancedejesus Jul 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

target.entity.id can be MV IIRC? Do we know if values are susceptible to change, causing unexpected events?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@terrancedejesus MV?

terrancedejesus
terrancedejesus reviewed Jul 14, 2025
View reviewed changes
rules/integrations/aws/exfiltration_dynamodb_table_exported_to_s3.toml
@@ -73,7 +68,7 @@ query = '''
event.dataset: "aws.cloudtrail"
and event.provider: "dynamodb.amazonaws.com"
and event.action: "ExportTableToPointInTime"
and aws.cloudtrail.flattened.request_parameters.dryRun: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of this was to ignore testing. ExportTableToPointInTime where successful if dryRun=True should signal testing but nothing was exported. May want to double check before you remove.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@terrancedejesus Were you actually able to run that command with dryRun parameter? I don't see that it's supported for this API call and I couldn't actually run the call with DryRun, that's why I removed it here. Did you have a different experience? I was using Python to run this script and tried adding that parameter but got "Unknown Parameter" error.

imays11
imays11 commented Jul 15, 2025
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Aegrah Aegrah Aegrah approved these changes

@terrancedejesus terrancedejesus terrancedejesus approved these changes

@Mikaayenson Mikaayenson Awaiting requested review from Mikaayenson

@w0rk3r w0rk3r Awaiting requested review from w0rk3r

@shashank-elastic shashank-elastic Awaiting requested review from shashank-elastic

Assignees

@imays11 imays11

Labels
backport: auto Domain: Cloud Integration: AWS AWS related rules Rule: Tuning tweaking or tuning an existing rule Team: TRADE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@imays11 @Aegrah @terrancedejesus @Mikaayenson