Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix ElasticSearch error with filter deletion #584

Merged
merged 19 commits into from
Feb 7, 2025

Conversation

antoinebhs
Copy link
Contributor

On a filter deletion with a lot of things to delete, ES returns a "All shards have failed" error.
That's because we are asking ElasticSearch to return all the equipment infos corresponding to deleted identifiables ids. If we have a lot of equipments deleted, the following list:
deletedEquipments.stream().map(EquipmentInfosToDelete::id).toList(),
can contain more than 10000 elements.
This equipment infos list is then used to avoid sending delete requests to ES for equipments that don't exist in the index. This is not necessary as ElasticSearch can handle this directly. In terms of performance, it's more efficient to send delete request with documents that don't exist instead of fetching the documents and deleting only the existing ones from my tests.

I added another constant for deletion partitioning as the existing one is too big for deletions. If there are too many ids in the request, the number of clauses is too large and can't be handled by ElasticSearch. The error looks like:

            "reason" : "maxClauseCount is set to 2114"

on my local deployment.
From the documentation, in the version we use there is a heuristic to determine the maxClauseCount from the available memory/heap (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-settings.html).
I set it by default to 2048 to be below the limit in our local deployment but we could override this value in other environment if we're looking to improve performance of delete with a lot of values.

Signed-off-by: BOUHOURS Antoine <[email protected]>
@antoinebhs antoinebhs force-pushed the fix-delete-by-filter-es branch from e583e4d to a8f2521 Compare December 26, 2024 16:02
@antoinebhs
Copy link
Contributor Author

antoinebhs commented Dec 26, 2024

Looks like we still see deleted equipments in search
Fixed in gridsuite/study-server#662

sBouzols and others added 7 commits January 17, 2025 16:36
… number of substations containing an equipment deletion is upper or equal to collectionThreshold

Signed-off-by: sBouzols <[email protected]>
Signed-off-by: BOUHOURS Antoine <[email protected]>
Signed-off-by: BOUHOURS Antoine <[email protected]>
@antoinebhs antoinebhs changed the base branch from main to fix_byFilter_deletion_impacts_reduction February 6, 2025 07:58
Signed-off-by: BOUHOURS Antoine <[email protected]>
Copy link
Contributor

@sBouzols sBouzols left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cpde Review OK
Tests OK

Signed-off-by: sBouzols <[email protected]>
Signed-off-by: BOUHOURS Antoine <[email protected]>
Slimane AMAR and others added 4 commits February 7, 2025 15:42
Review 1
Signed-off-by: Slimane AMAR <[email protected]>
Review 2
Signed-off-by: Slimane AMAR <[email protected]>
Base automatically changed from fix_byFilter_deletion_impacts_reduction to main February 7, 2025 15:17
…ion-server into fix-delete-by-filter-es
@antoinebhs antoinebhs merged commit 736f727 into main Feb 7, 2025
4 checks passed
@antoinebhs antoinebhs deleted the fix-delete-by-filter-es branch February 7, 2025 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants