detect/entropy: Add entropy keyword #12542
Open
+1,051
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue: 4162 This commit adds a floating-point analog to `uint.rs` targeted initially to be used by the forthcoming entropy keyword.
This commit adds
- Parser for the entropy keyword
- Calculation of content the Shannon entropy value
Issue: 4162
The entropy keyword syntax is the keyword entropy followed by options
and the entropy value for comparison.
The minimum entropy keyword specification is:
entropy: value <entropy-spec>
This results in the calculated entropy value being compared with
<entropy-spec> with the equality operator.
A match occurs when the values and operator agree. This example matches
if the calculated and entropy value are the same.
When entropy keyword options are specified, all options and "value" must
be comma-separated. Options and value may be specified in any order.
Options have default values:
- bytes is equal to the current content length
- offset is 0
- comparison with value is equality
entropy: [bytes <byteval>] [offset <offsetval>] value <entropy-spec>
Using default values:
entropy: bytes 0, offset 0, value =<entropy-spec>
<entropy-spec> is: <operator> (see below) and a value, e.g., "< 4.1"
The following operators are available from the float crate:
- = (default): Match when calculated entropy value equals specified entropy value
- < Match when calculated entropy value is strictly less than specified entropy value
- <= Match when calculated entropy value is less than or equal to specified entropy value
- > Match when calculated entropy value is strictly greater than specified entropy value
- >= Match when calculated entropy value is greater than or equal to specified entropy value
- != Match when calculated entropy value is not equal to specified entropy value
- x-y Match when calculated entropy value is in the range, exclusive
- !x-y Match when calculated entropy value is not in the range, exclusive
This commit adds keyword/build support for the entropy keyword. The entropy keyword compares an entropy value with a value calculated according to the Shannon entropy on the available content. Issue: 4162
This commit causes the content inspection engine to recognize and invoke the entropy "match" function when the entropy keyword is used. Issue: 4162
This commit updates the - Upgrade notes for 7 to 8 - Payload keyword section Both are update to document the new entropy keyword.
jlucovsky
requested review from
jasonish,
jufajardini and
victorjulien
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12542 +/- ##
==========================================
+ Coverage 80.68% 80.73% +0.04%
==========================================
Files 925 928 +3
Lines 258914 259518 +604
==========================================
+ Hits 208914 209516 +602
- Misses 50000 50002 +2
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
WARNING:
Pipeline 24668 |
jasonish
requested changes
Feb 7, 2025
Comment on lines
+33
to
+40
| #[repr(C)] | ||
| #[derive(Debug)] | ||
| pub struct DetectEntropyData { | ||
| offset: i32, | ||
| nbytes: i32, | ||
| value: DetectFloatData<f64>, | ||
| flags: u8, | ||
| } |
There was a problem hiding this comment.
This looks like its used opaquely from C? If so, we can drop the repr here to avoid providing the full structure to C, which IMO is a good idea unless C needs to get at the internals.
There was a problem hiding this comment.
Thanks -- will do.
Comment on lines
+143
to
+149
| fn calculate_entropy(data: *const u8, length: i32) -> f64 { | ||
| if data.is_null() || length <= 0 { | ||
| return 0.0; | ||
| } | ||
|
|
||
| // Convert the raw pointer to a slice safely | ||
| let data_slice = unsafe { slice::from_raw_parts(data, length as usize) }; |
There was a problem hiding this comment.
This conversion should be done in the calling function, then it can just pass the &[u8] here without length and avoid the unsafe here, since SCDetectEntropyMatch is already unsafe. Also makes its more usable from Rust if ever needed.
There was a problem hiding this comment.
Good point. I'll make this change.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continuation of #12539
The entropy keyword syntax is the keyword
entropyfollowed by optionsand the entropy value for comparison.
The minimum entropy keyword specification is:
entropy: value <entropy-spec>This results in the calculated entropy value being compared with
with the equality operator.
A match occurs when the values and operator agree. This example matches
if the calculated and entropy value are the same.
When entropy keyword options are specified, all options and "value" must
be comma-separated. Options and value may be specified in any order.
Options have default values:
entropy: [bytes <byteval>] [offset <offsetval>] value <entropy-spec>Using default values:
entropy: bytes 0, offset 0, value =<entropy-spec>is: (see below) and a value, e.g., "< 4.1"
The following operators are available from the float crate introduced with this pr:
- = (default): Match when calculated entropy value equals specified entropy value
- < Match when calculated entropy value is strictly less than specified entropy value
- <= Match when calculated entropy value is less than or equal to the specified entropy value
- > Match when calculated entropy value is strictly greater than specified entropy value
- >= Match when calculated entropy value is greater than or equal to the specified entropy value
- != Match when the calculated entropy value is not equal to the specified entropy value
- x-y Match when calculated entropy value is in the range, exclusive
- !x-y Match when calculated entropy value is not in the range, exclusive
Link to ticket: https://redmine.openinfosecfoundation.org/issues/4162
Describe changes:
Updates:
Provide values to any of the below to override the defaults.
link to the pull request in the respective
_BRANCHvariable.SV_BRANCH=OISF/suricata-verify#2232