Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add quick filters on scenarios, simulations and atomic testing #1352

Merged
merged 7 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,10 @@
scenario.setSubtitle(scenarioNode.get("scenario_subtitle").textValue());
scenario.setCategory(scenarioNode.get("scenario_category").textValue());
scenario.setMainFocus(scenarioNode.get("scenario_main_focus").textValue());
scenario.setSeverity(scenarioNode.get("scenario_severity").textValue());
if (scenarioNode.get("scenario_severity") != null) {
String severity = scenarioNode.get("scenario_severity").textValue();
scenario.setSeverity(Scenario.SEVERITY.valueOf(severity));

Check warning on line 289 in openbas-api/src/main/java/io/openbas/importer/V1_DataImporter.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/importer/V1_DataImporter.java#L288-L289

Added lines #L288 - L289 were not covered by tests
}
if (scenarioNode.get("scenario_recurrence") != null) {
scenario.setRecurrence(scenarioNode.get("scenario_recurrence").textValue());
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
import java.io.InputStream;
import java.time.Instant;
import java.util.*;
import java.util.function.UnaryOperator;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.stream.Collectors;
Expand All @@ -53,7 +52,6 @@
import static io.openbas.database.specification.ExerciseSpecification.findGrantedFor;
import static io.openbas.helper.StreamHelper.fromIterable;
import static io.openbas.helper.StreamHelper.iterableToSet;
import static io.openbas.rest.exercise.utils.ExerciseUtils.handleCustomFilter;
import static io.openbas.service.ImportService.EXPORT_ENTRY_ATTACHMENT;
import static io.openbas.service.ImportService.EXPORT_ENTRY_EXERCISE;
import static io.openbas.utils.pagination.PaginationUtils.buildPaginationCriteriaBuilder;
Expand Down Expand Up @@ -644,23 +642,16 @@

@PostMapping(EXERCISE_URI + "/search")
public Page<ExerciseSimple> exercises(@RequestBody @Valid final SearchPaginationInput searchPaginationInput) {
UnaryOperator<Specification<Exercise>> finalSpecification = handleCustomFilter(
searchPaginationInput
);

if (currentUser().isAdmin()) {
return buildPaginationCriteriaBuilder(
(Specification<Exercise> specification, Pageable pageable) -> this.exerciseService.exercises(
finalSpecification.apply(specification),
pageable
),
this.exerciseService::exercises,
searchPaginationInput,
Exercise.class
);
} else {
return buildPaginationCriteriaBuilder(
(Specification<Exercise> specification, Pageable pageable) -> this.exerciseService.exercises(
finalSpecification.apply(findGrantedFor(currentUser().getId()).and(specification)),
findGrantedFor(currentUser().getId()).and(specification),

Check warning on line 654 in openbas-api/src/main/java/io/openbas/rest/exercise/ExerciseApi.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/exercise/ExerciseApi.java#L654

Added line #L654 was not covered by tests
pageable
),
searchPaginationInput,
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package io.openbas.rest.inject;

import io.openbas.database.model.Inject;
import io.openbas.database.model.InjectTestStatus;
import io.openbas.database.specification.InjectSpecification;
import io.openbas.rest.helper.RestBehavior;
import io.openbas.rest.inject.output.InjectOutput;
import io.openbas.service.InjectService;
Expand All @@ -16,13 +16,17 @@
import jakarta.validation.constraints.NotBlank;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.*;

import java.util.List;

import static io.openbas.database.specification.InjectSpecification.fromExercise;
import static io.openbas.rest.exercise.ExerciseApi.EXERCISE_URI;
import static io.openbas.utils.pagination.PaginationUtils.buildPaginationCriteriaBuilder;

@RestController
@RequiredArgsConstructor
Expand All @@ -45,7 +49,22 @@
@PreAuthorize("isExerciseObserver(#exerciseId)")
@Transactional(readOnly = true)
public Iterable<InjectOutput> exerciseInjectsSimple(@PathVariable @NotBlank final String exerciseId) {
return injectService.injects(InjectSpecification.fromExercise(exerciseId));
return injectService.injects(fromExercise(exerciseId));

Check warning on line 52 in openbas-api/src/main/java/io/openbas/rest/inject/ExerciseInjectApi.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/inject/ExerciseInjectApi.java#L52

Added line #L52 was not covered by tests
}

@PostMapping(EXERCISE_URI + "/{exerciseId}/injects/simple")
@PreAuthorize("isExerciseObserver(#exerciseId)")
@Transactional(readOnly = true)
public Iterable<InjectOutput> exerciseInjectsSimple(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it normal you keep the same API with pagination and not with pagination ? Can you merge the two or there is a reason that we keep the old one ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally the API endpoints of all entities returned the entire DB table and everything was stored at the store level.

Going through pagination improves performance to return a more restricted list.

Unfortunately, there is still a lot of code to migrate to use the new endpoints and not all of it can be done in this task. But it’s something to keep in mind!

@PathVariable @NotBlank final String exerciseId,
@RequestBody @Valid final SearchPaginationInput searchPaginationInput) {
return buildPaginationCriteriaBuilder(
(Specification<Inject> specification, Pageable pageable) -> this.injectService.injects(
fromExercise(exerciseId).and(specification), pageable

Check warning on line 63 in openbas-api/src/main/java/io/openbas/rest/inject/ExerciseInjectApi.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/inject/ExerciseInjectApi.java#L61-L63

Added lines #L61 - L63 were not covered by tests
),
searchPaginationInput,
Inject.class
);
}

@DeleteMapping(EXERCISE_URI + "/{exerciseId}/injects")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package io.openbas.rest.inject;

import io.openbas.database.model.Inject;
import io.openbas.database.model.InjectTestStatus;
import io.openbas.database.specification.InjectSpecification;
import io.openbas.rest.helper.RestBehavior;
import io.openbas.rest.inject.output.InjectOutput;
import io.openbas.service.InjectService;
Expand All @@ -11,13 +11,17 @@
import jakarta.validation.constraints.NotBlank;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.*;

import java.util.List;

import static io.openbas.database.specification.InjectSpecification.fromScenario;
import static io.openbas.rest.scenario.ScenarioApi.SCENARIO_URI;
import static io.openbas.utils.pagination.PaginationUtils.buildPaginationCriteriaBuilder;

@RestController
@RequiredArgsConstructor
Expand All @@ -31,7 +35,22 @@
@PreAuthorize("isScenarioObserver(#scenarioId)")
@Transactional(readOnly = true)
public Iterable<InjectOutput> scenarioInjectsSimple(@PathVariable @NotBlank final String scenarioId) {
return injectService.injects(InjectSpecification.fromScenario(scenarioId));
return injectService.injects(fromScenario(scenarioId));

Check warning on line 38 in openbas-api/src/main/java/io/openbas/rest/inject/ScenarioInjectApi.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/inject/ScenarioInjectApi.java#L38

Added line #L38 was not covered by tests
}

@PostMapping(SCENARIO_URI + "/{scenarioId}/injects/simple")
@PreAuthorize("isScenarioObserver(#scenarioId)")
@Transactional(readOnly = true)
public Iterable<InjectOutput> scenarioInjectsSimple(
@PathVariable @NotBlank final String scenarioId,
@RequestBody @Valid final SearchPaginationInput searchPaginationInput) {
return buildPaginationCriteriaBuilder(
(Specification<Inject> specification, Pageable pageable) -> this.injectService.injects(
fromScenario(scenarioId).and(specification), pageable

Check warning on line 49 in openbas-api/src/main/java/io/openbas/rest/inject/ScenarioInjectApi.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/inject/ScenarioInjectApi.java#L47-L49

Added lines #L47 - L49 were not covered by tests
),
searchPaginationInput,
Inject.class
);
}

@DeleteMapping(SCENARIO_URI + "/{scenarioId}/injects")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,14 @@
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.security.access.annotation.Secured;
import org.springframework.web.bind.annotation.*;

import java.time.Instant;
import java.util.function.UnaryOperator;

import static io.openbas.database.model.User.ROLE_ADMIN;
import static io.openbas.helper.DatabaseHelper.updateRelation;
import static io.openbas.helper.StreamHelper.fromIterable;
import static io.openbas.rest.injector_contract.utils.InjectorContractUtils.handleCustomFilter;
import static io.openbas.utils.pagination.PaginationUtils.buildPaginationCriteriaBuilder;

@RequiredArgsConstructor
Expand All @@ -49,15 +45,8 @@ public Iterable<RawInjectorsContrats> injectContracts() {

@PostMapping("/api/injector_contracts/search")
public Page<InjectorContractOutput> injectorContracts(@RequestBody @Valid final SearchPaginationInput searchPaginationInput) {
UnaryOperator<Specification<InjectorContract>> finalSpecification = handleCustomFilter(
searchPaginationInput
);

return buildPaginationCriteriaBuilder(
(Specification<InjectorContract> specification, Pageable pageable) -> this.injectorContractService.injectorContracts(
finalSpecification.apply(specification),
pageable
),
this.injectorContractService::injectorContracts,
searchPaginationInput,
InjectorContract.class
);
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,6 @@ public void setDocumentRepository(DocumentRepository documentRepository) {
this.documentRepository = documentRepository;
}

@GetMapping("/api/payloads")
public Iterable<Payload> payloads() {
return payloadRepository.findAll();
}

@PostMapping("/api/payloads/search")
public Page<Payload> payloads(@RequestBody @Valid final SearchPaginationInput searchPaginationInput) {
return buildPaginationJPA(
Expand Down
Original file line number Diff line number Diff line change
@@ -1,36 +1,57 @@
package io.openbas.rest.scenario.utils;

import io.openbas.database.model.Filters;
import io.openbas.database.model.Scenario;
import io.openbas.utils.CustomFilterUtils;
import io.openbas.database.specification.ScenarioSpecification;
import io.openbas.utils.pagination.SearchPaginationInput;
import org.jetbrains.annotations.NotNull;
import org.springframework.data.jpa.domain.Specification;

import java.util.Collections;
import java.util.Map;
import java.util.Optional;
import java.util.function.Function;
import java.util.function.UnaryOperator;

public class ScenarioUtils {
import static io.openbas.utils.CustomFilterUtils.computeMode;
import static java.util.Optional.ofNullable;

private static final String SCENARIO_KILL_CHAIN_PHASES_FILTER = "scenario_kill_chain_phases";
private static final Map<String, String> CORRESPONDENCE_MAP = Collections.singletonMap(
SCENARIO_KILL_CHAIN_PHASES_FILTER, "injects.injectorContract.attackPatterns.killChainPhases.id"
);
public class ScenarioUtils {

private ScenarioUtils() {

}

private static final String SCENARIO_RECURRENCE_FILTER = "scenario_recurrence";

/**
* Manage filters that are not directly managed by the generic mechanics -> scenario_kill_chain_phases
*/
public static UnaryOperator<Specification<Scenario>> handleCustomFilter(
public static Function<Specification<Scenario>, Specification<Scenario>> handleDeepFilter(
@NotNull final SearchPaginationInput searchPaginationInput) {
return handleCustomFilter(searchPaginationInput);
}

private static UnaryOperator<Specification<Scenario>> handleCustomFilter(
@NotNull final SearchPaginationInput searchPaginationInput) {
return CustomFilterUtils.handleCustomFilter(
searchPaginationInput,
SCENARIO_KILL_CHAIN_PHASES_FILTER,
CORRESPONDENCE_MAP
);
// Existence of the filter
Optional<Filters.Filter> scenarioRecurrenceFilterOpt = ofNullable(searchPaginationInput.getFilterGroup())
.flatMap(f -> f.findByKey(SCENARIO_RECURRENCE_FILTER));

if (scenarioRecurrenceFilterOpt.isPresent()) {
// Purge filter
searchPaginationInput.getFilterGroup().removeByKey(SCENARIO_RECURRENCE_FILTER);
Specification<Scenario> customSpecification = null;

Check warning on line 42 in openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java#L41-L42

Added lines #L41 - L42 were not covered by tests
if (scenarioRecurrenceFilterOpt.get().getValues().contains("Scheduled")) {
customSpecification = ScenarioSpecification.isRecurring();

Check warning on line 44 in openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java#L44

Added line #L44 was not covered by tests
} else if (scenarioRecurrenceFilterOpt.get().getValues().contains("Not planned")) {
customSpecification = ScenarioSpecification.noRecurring();

Check warning on line 46 in openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java#L46

Added line #L46 was not covered by tests
}
if (customSpecification != null) {
return computeMode(searchPaginationInput, customSpecification);

Check warning on line 49 in openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java#L49

Added line #L49 was not covered by tests
}
return (Specification<Scenario> specification) -> specification;

Check warning on line 51 in openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/rest/scenario/utils/ScenarioUtils.java#L51

Added line #L51 was not covered by tests
} else {
return (Specification<Scenario> specification) -> specification;
}
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,7 @@

// -- PAGINATION --

public Page<AtomicTestingOutput> findAllAtomicTestings(SearchPaginationInput searchPaginationInput) {
public Page<AtomicTestingOutput> findAllAtomicTestings(@NotNull final SearchPaginationInput searchPaginationInput) {
Specification<Inject> customSpec = Specification.where((root, query, cb) -> {
Predicate predicate = cb.conjunction();
predicate = cb.and(predicate, cb.isNull(root.get("scenario")));
Expand All @@ -275,7 +275,7 @@
});
return buildPaginationCriteriaBuilder(
(Specification<Inject> specification, Pageable pageable) -> this.atomicTestings(
specification.and(customSpec), pageable),
customSpec.and(specification), pageable),

Check warning on line 278 in openbas-api/src/main/java/io/openbas/service/AtomicTestingService.java

View check run for this annotation

Codecov / codecov/patch

openbas-api/src/main/java/io/openbas/service/AtomicTestingService.java#L278

Added line #L278 was not covered by tests
searchPaginationInput,
Inject.class
);
Expand Down
Loading