Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature findings api enhancements #914

Merged
merged 32 commits into from
Mar 13, 2024
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
2ffe265
get all findings as part of findings API enhancement
riysaxen-amzn Jan 16, 2024
705988f
findingsAPI feature enhancements (address comments to prev PR)
riysaxen-amzn Jan 19, 2024
854c44c
findingsAPI feature enhancements (address comments to prev PR)
riysaxen-amzn Jan 19, 2024
8bb1d7e
added support for param in Finding API
riysaxen-amzn Jan 31, 2024
303493c
added detectionType as param for Findings API enhancements
riysaxen-amzn Jan 31, 2024
d6a914e
added few tests to validate findings by params
riysaxen-amzn Feb 14, 2024
d8d4051
Merge branch 'opensearch-project:main' into feature-findingsAPIenhanc…
riysaxen-amzn Feb 14, 2024
0eb7dce
added test for searchString param in FindingsAPI
riysaxen-amzn Feb 24, 2024
6b2a15c
adding addiional params findingIds, startTime and endTime as findings…
riysaxen-amzn Feb 29, 2024
0804c8b
added params in getFindingsByDetectorId func
riysaxen-amzn Mar 5, 2024
b9349c1
changed the startTime and endTime req input format
riysaxen-amzn Mar 6, 2024
5a80380
Merge branch 'main' into feature-findingsAPIenhancements
riysaxen-amzn Mar 9, 2024
13f74db
fix merge conflixt
riysaxen-amzn Mar 9, 2024
02f7000
fix integ test failures in findings API
riysaxen-amzn Mar 11, 2024
8a7c3d8
fix integ tests
riysaxen-amzn Mar 11, 2024
b29cd33
refactored the logic
riysaxen-amzn Mar 13, 2024
7a63ec0
remove unused imports
riysaxen-amzn Mar 13, 2024
ccc84ea
address the pr comments
riysaxen-amzn Mar 13, 2024
dae015b
address pr comments
riysaxen-amzn Mar 13, 2024
b45660b
SA integ tests fix
riysaxen-amzn Mar 13, 2024
5ba7a0f
SA integ tests fix
riysaxen-amzn Mar 13, 2024
a5adf95
Merge branch 'main' into feature-findingsAPIenhancements
riysaxen-amzn Mar 13, 2024
369ca7d
fix integ tests for findings
sbcd90 Mar 11, 2024
c15e577
fix conflixt errors
riysaxen-amzn Mar 13, 2024
02fc845
fix conflixt errors
riysaxen-amzn Mar 13, 2024
de8bf2f
fix conflixt errors
riysaxen-amzn Mar 13, 2024
2587fca
fix conflixt errors
riysaxen-amzn Mar 13, 2024
f3eb85d
fix integ tests
riysaxen-amzn Mar 13, 2024
f20ed7f
fix integ tests
riysaxen-amzn Mar 13, 2024
8fc5960
fix integ tests
riysaxen-amzn Mar 13, 2024
19ff06e
fix flaky integ tests
riysaxen-amzn Mar 13, 2024
d8043ef
address pr comments
riysaxen-amzn Mar 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added few tests to validate findings by params
Signed-off-by: Riya Saxena <[email protected]>
riysaxen-amzn committed Feb 14, 2024
commit d6a914eb41cac31ad642858a9e002d36b26a91eb
30 changes: 30 additions & 0 deletions src/test/java/org/opensearch/securityanalytics/TestHelpers.java
Original file line number Diff line number Diff line change
@@ -259,6 +259,36 @@ public static String randomRule() {
"level: high";
}

public static String randomRuleWithCriticalSeverity() {
return "title: Remote Encrypting File System Abuse\n" +
"id: 5f92fff9-82e2-48eb-8fc1-8b133556a551\n" +
"description: Detects remote RPC calls to possibly abuse remote encryption service via MS-EFSR\n" +
"references:\n" +
" - https://attack.mitre.org/tactics/TA0008/\n" +
" - https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-36942\n" +
" - https://github.com/jsecurity101/MSRPC-to-ATTACK/blob/main/documents/MS-EFSR.md\n" +
" - https://github.com/zeronetworks/rpcfirewall\n" +
" - https://zeronetworks.com/blog/stopping_lateral_movement_via_the_rpc_firewall/\n" +
"tags:\n" +
" - attack.defense_evasion\n" +
"status: experimental\n" +
"author: Sagie Dulce, Dekel Paz\n" +
"date: 2022/01/01\n" +
"modified: 2022/01/01\n" +
"logsource:\n" +
" product: rpc_firewall\n" +
" category: application\n" +
" definition: 'Requirements: install and apply the RPC Firewall to all processes with \"audit:true action:block uuid:df1941c5-fe89-4e79-bf10-463657acf44d or c681d488-d850-11d0-8c52-00c04fd90f7e'\n" +
"detection:\n" +
" selection:\n" +
" EventID: 22\n" +
" condition: selection\n" +
"falsepositives:\n" +
" - Legitimate usage of remote file encryption\n" +
"level: critical";
}


public static String randomNullRule() {
return "title: null field\n" +
"id: 5f92fff9-82e2-48eb-8fc1-8b133556a551\n" +
Original file line number Diff line number Diff line change
@@ -30,12 +30,8 @@
import org.opensearch.securityanalytics.model.DetectorRule;
import org.opensearch.securityanalytics.model.DetectorTrigger;

import static org.opensearch.securityanalytics.TestHelpers.netFlowMappings;
import static org.opensearch.securityanalytics.TestHelpers.randomDetectorType;
import static org.opensearch.securityanalytics.TestHelpers.randomDetectorWithTriggers;
import static org.opensearch.securityanalytics.TestHelpers.randomDoc;
import static org.opensearch.securityanalytics.TestHelpers.randomIndex;
import static org.opensearch.securityanalytics.TestHelpers.windowsIndexMapping;
import static java.util.Collections.emptyList;
import static org.opensearch.securityanalytics.TestHelpers.*;
import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_INDEX_MAX_AGE;
import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_MAX_DOCS;
import static org.opensearch.securityanalytics.settings.SecurityAnalyticsSettings.FINDING_HISTORY_RETENTION_PERIOD;
@@ -371,6 +367,234 @@ public void testGetAllFindings_success() throws IOException {
Assert.assertEquals(2, getFindingsBody.get("total_findings"));
}

public void testGetFindings_byDetectionType_success() throws IOException {
String index1 = createTestIndex(randomIndex(), windowsIndexMapping());

// Execute CreateMappingsAction to add alias mapping for index
Request createMappingRequest = new Request("POST", SecurityAnalyticsPlugin.MAPPER_BASE_URI);
// both req params and req body are supported
createMappingRequest.setJsonEntity(
"{ \"index_name\":\"" + index1 + "\"," +
" \"rule_topic\":\"" + randomDetectorType() + "\", " +
" \"partial\":true" +
"}"
);

Response response = client().performRequest(createMappingRequest);
assertEquals(HttpStatus.SC_OK, response.getStatusLine().getStatusCode());

// index 2
String index2 = createTestIndex("netflow_test", netFlowMappings());

// Execute CreateMappingsAction to add alias mapping for index
createMappingRequest = new Request("POST", SecurityAnalyticsPlugin.MAPPER_BASE_URI);
// both req params and req body are supported
createMappingRequest.setJsonEntity(
"{ \"index_name\":\"" + index2 + "\"," +
" \"rule_topic\":\"netflow\", " +
" \"partial\":true" +
"}"
);

response = client().performRequest(createMappingRequest);
assertEquals(HttpStatus.SC_OK, response.getStatusLine().getStatusCode());
// Detector 1 - WINDOWS
String randomDocRuleId = createRule(randomRule());
List<DetectorRule> detectorRules = List.of(new DetectorRule(randomDocRuleId));
DetectorInput input = new DetectorInput("windows detector for security analytics", List.of("windows"), detectorRules,
emptyList());
Detector detector1 = randomDetectorWithInputsAndThreatIntel(List.of(input), true);

Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector1));
Assert.assertEquals("Create detector failed", RestStatus.CREATED, restStatus(createResponse));

Map<String, Object> responseBody = asMap(createResponse);
String createdId = responseBody.get("_id").toString();

String request = "{\n" +
" \"query\" : {\n" +
" \"match\":{\n" +
" \"_id\": \"" + createdId + "\"\n" +
" }\n" +
" }\n" +
"}";
List<SearchHit> hits = executeSearch(Detector.DETECTORS_INDEX, request);
SearchHit hit = hits.get(0);
String monitorId1 = ((List<String>) ((Map<String, Object>) hit.getSourceAsMap().get("detector")).get("monitor_id")).get(0);
// Detector 2 - NETWORK
DetectorInput inputNetflow = new DetectorInput("windows detector for security analytics", List.of("netflow_test"), Collections.emptyList(),
getPrePackagedRules("network").stream().map(DetectorRule::new).collect(Collectors.toList()));
Detector detector2 = randomDetectorWithTriggers(
getPrePackagedRules("network"),
List.of(new DetectorTrigger(null, "test-trigger", "1", List.of("network"), List.of(), List.of(), List.of(), List.of(), List.of())),
"network",
inputNetflow
);

createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector2));
Assert.assertEquals("Create detector failed", RestStatus.CREATED, restStatus(createResponse));

responseBody = asMap(createResponse);

createdId = responseBody.get("_id").toString();

request = "{\n" +
" \"query\" : {\n" +
" \"match\":{\n" +
" \"_id\": \"" + createdId + "\"\n" +
" }\n" +
" }\n" +
"}";
hits = executeSearch(Detector.DETECTORS_INDEX, request);
hit = hits.get(0);
String monitorId2 = ((List<String>) ((Map<String, Object>) hit.getSourceAsMap().get("detector")).get("monitor_id")).get(0);

indexDoc(index1, "1", randomDoc());
indexDoc(index2, "1", randomDoc());
// execute monitor 1
Response executeResponse = executeAlertingMonitor(monitorId1, Collections.emptyMap());
Map<String, Object> executeResults = entityAsMap(executeResponse);

int noOfSigmaRuleMatches = ((List<Map<String, Object>>) ((Map<String, Object>) executeResults.get("input_results")).get("results")).get(0).size();
Assert.assertEquals(1, noOfSigmaRuleMatches);

// execute monitor 2
executeResponse = executeAlertingMonitor(monitorId2, Collections.emptyMap());
executeResults = entityAsMap(executeResponse);

noOfSigmaRuleMatches = ((List<Map<String, Object>>) ((Map<String, Object>) executeResults.get("input_results")).get("results")).get(0).size();
Assert.assertEquals(1, noOfSigmaRuleMatches);

// client().performRequest(new Request("POST", "_refresh"));

// Call GetFindings API for first detector by detectionType
Map<String, String> params = new HashMap<>();
params.put("detectionType", "rule");
Response getFindingsResponse = makeRequest(client(), "GET", SecurityAnalyticsPlugin.FINDINGS_BASE_URI + "/_search", params, null);
Map<String, Object> getFindingsBody = entityAsMap(getFindingsResponse);
Assert.assertEquals(2, getFindingsBody.get("total_findings"));
}

public void testGetFindings_bySeverity_success() throws IOException {
String index1 = createTestIndex(randomIndex(), windowsIndexMapping());

// Execute CreateMappingsAction to add alias mapping for index
Request createMappingRequest = new Request("POST", SecurityAnalyticsPlugin.MAPPER_BASE_URI);
// both req params and req body are supported
createMappingRequest.setJsonEntity(
"{ \"index_name\":\"" + index1 + "\"," +
" \"rule_topic\":\"" + randomDetectorType() + "\", " +
" \"partial\":true" +
"}"
);

Response response = client().performRequest(createMappingRequest);
assertEquals(HttpStatus.SC_OK, response.getStatusLine().getStatusCode());

// index 2
String index2 = createTestIndex("windows1", windowsIndexMapping());

// Execute CreateMappingsAction to add alias mapping for index
createMappingRequest = new Request("POST", SecurityAnalyticsPlugin.MAPPER_BASE_URI);
// both req params and req body are supported
createMappingRequest.setJsonEntity(
"{ \"index_name\":\"" + index2 + "\"," +
" \"rule_topic\":\"windows\", " +
" \"partial\":true" +
"}"
);

response = client().performRequest(createMappingRequest);
assertEquals(HttpStatus.SC_OK, response.getStatusLine().getStatusCode());
// Detector 1 - WINDOWS
String randomDocRuleId = createRule(randomRule());
List<DetectorRule> detectorRules = List.of(new DetectorRule(randomDocRuleId));
DetectorInput input = new DetectorInput("windows detector for security analytics", List.of("windows"), detectorRules,
emptyList());
Detector detector1 = randomDetectorWithTriggers(
getPrePackagedRules("windows"),
List.of(new DetectorTrigger(null, "test-trigger", "1", List.of("windows"), List.of(), List.of(), List.of(), List.of(), List.of())),
"windows",
input
);

Response createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector1));
Assert.assertEquals("Create detector failed", RestStatus.CREATED, restStatus(createResponse));

Map<String, Object> responseBody = asMap(createResponse);
String createdId = responseBody.get("_id").toString();

String request = "{\n" +
" \"query\" : {\n" +
" \"match\":{\n" +
" \"_id\": \"" + createdId + "\"\n" +
" }\n" +
" }\n" +
"}";
List<SearchHit> hits = executeSearch(Detector.DETECTORS_INDEX, request);
SearchHit hit = hits.get(0);
String monitorId1 = ((List<String>) ((Map<String, Object>) hit.getSourceAsMap().get("detector")).get("monitor_id")).get(0);
// Detector 2 - CRITICAL Severity Netflow
String randomDocRuleId2 = createRule(randomRuleWithCriticalSeverity());
List<DetectorRule> detectorRules2 = List.of(new DetectorRule(randomDocRuleId2));
DetectorInput inputNetflow = new DetectorInput("windows detector for security analytics", List.of("windows"), detectorRules2,
emptyList());
Detector detector2 = randomDetectorWithTriggers(
getPrePackagedRules("windows1"),
List.of(new DetectorTrigger(null, "test-trigger", "0", List.of("windows1"), List.of(), List.of(), List.of(), List.of(), List.of())),
"windows",
inputNetflow
);

createResponse = makeRequest(client(), "POST", SecurityAnalyticsPlugin.DETECTOR_BASE_URI, Collections.emptyMap(), toHttpEntity(detector2));
Assert.assertEquals("Create detector failed", RestStatus.CREATED, restStatus(createResponse));

responseBody = asMap(createResponse);
logger.info("Created response 2 : {}", responseBody.toString());

createdId = responseBody.get("_id").toString();

request = "{\n" +
" \"query\" : {\n" +
" \"match\":{\n" +
" \"_id\": \"" + createdId + "\"\n" +
" }\n" +
" }\n" +
"}";
hits = executeSearch(Detector.DETECTORS_INDEX, request);
hit = hits.get(0);
String monitorId2 = ((List<String>) ((Map<String, Object>) hit.getSourceAsMap().get("detector")).get("monitor_id")).get(0);

indexDoc(index1, "1", randomDoc());
indexDoc(index2, "2", randomDoc());
// execute monitor 1
Response executeResponse = executeAlertingMonitor(monitorId1, Collections.emptyMap());
Map<String, Object> executeResults = entityAsMap(executeResponse);
int noOfSigmaRuleMatches = ((List<Map<String, Object>>) ((Map<String, Object>) executeResults.get("input_results")).get("results")).get(0).size();
Assert.assertEquals(1, noOfSigmaRuleMatches);

// execute monitor 2
executeResponse = executeAlertingMonitor(monitorId2, Collections.emptyMap());
executeResults = entityAsMap(executeResponse);
noOfSigmaRuleMatches = ((List<Map<String, Object>>) ((Map<String, Object>) executeResults.get("input_results")).get("results")).get(0).size();
Assert.assertEquals(1, noOfSigmaRuleMatches);

client().performRequest(new Request("POST", "_refresh"));

// Call GetFindings API for first detector by severity
Map<String, String> params = new HashMap<>();
params.put("severity", "high");
Response getFindingsResponse = makeRequest(client(), "GET", SecurityAnalyticsPlugin.FINDINGS_BASE_URI + "/_search", params, null);
Map<String, Object> getFindingsBody = entityAsMap(getFindingsResponse);
Assert.assertEquals(1, getFindingsBody.get("total_findings"));
// Call GetFindings API for second detector by severity
params.clear();
params.put("severity", "critical");
getFindingsResponse = makeRequest(client(), "GET", SecurityAnalyticsPlugin.FINDINGS_BASE_URI + "/_search", params, null);
getFindingsBody = entityAsMap(getFindingsResponse);
Assert.assertEquals(1, getFindingsBody.get("total_findings"));
}

public void testGetFindings_rolloverByMaxAge_success() throws IOException, InterruptedException {

updateClusterSetting(FINDING_HISTORY_ROLLOVER_PERIOD.getKey(), "1s");
Original file line number Diff line number Diff line change
@@ -142,7 +142,7 @@ public void testGetFindings_success() {
ActionListener l = invocation.getArgument(4);
l.onResponse(getFindingsResponse);
return null;
}).when(findingsService).getFindingsByMonitorIds(any(), any(), anyString(), any(Table.class), any(ActionListener.class));
}).when(findingsService).getFindingsByMonitorIds(any(), any(), anyString(), any(Table.class), anyString(), anyString(), any(ActionListener.class));

// Call getFindingsByDetectorId
Table table = new Table(
@@ -209,7 +209,7 @@ public void testGetFindings_getFindingsByMonitorIdFailure() {
ActionListener l = invocation.getArgument(4);
l.onFailure(new IllegalArgumentException("Error getting findings"));
return null;
}).when(findingsService).getFindingsByMonitorIds(any(), any(), anyString(), any(Table.class), any(ActionListener.class));
}).when(findingsService).getFindingsByMonitorIds(any(), any(), anyString(), any(Table.class), anyString(), anyString(), any(ActionListener.class));

// Call getFindingsByDetectorId
Table table = new Table(
Original file line number Diff line number Diff line change
@@ -31,12 +31,7 @@
import java.util.Map;
import java.util.stream.Collectors;

import static org.opensearch.securityanalytics.TestHelpers.netFlowMappings;
import static org.opensearch.securityanalytics.TestHelpers.randomDetectorType;
import static org.opensearch.securityanalytics.TestHelpers.randomDetectorWithTriggers;
import static org.opensearch.securityanalytics.TestHelpers.randomDoc;
import static org.opensearch.securityanalytics.TestHelpers.randomIndex;
import static org.opensearch.securityanalytics.TestHelpers.windowsIndexMapping;
import static org.opensearch.securityanalytics.TestHelpers.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plz revert

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed


public class SecureFindingRestApiIT extends SecurityAnalyticsRestTestCase {