-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ElasticSearch error with filter deletion #584
Conversation
Signed-off-by: BOUHOURS Antoine <[email protected]>
e583e4d
to
a8f2521
Compare
|
Looks like we still see deleted equipments in search |
… number of substations containing an equipment deletion is upper or equal to collectionThreshold Signed-off-by: sBouzols <[email protected]>
Signed-off-by: BOUHOURS Antoine <[email protected]>
Signed-off-by: BOUHOURS Antoine <[email protected]>
…hub.com/gridsuite/network-modification-server into fix-delete-by-filter-es
Signed-off-by: BOUHOURS Antoine <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cpde Review OK
Tests OK
Signed-off-by: sBouzols <[email protected]>
src/main/java/org/gridsuite/modification/server/elasticsearch/EquipmentInfosService.java
Outdated
Show resolved
Hide resolved
Signed-off-by: BOUHOURS Antoine <[email protected]>
Signed-off-by: Slimane AMAR <[email protected]>
…hub.com/gridsuite/network-modification-server into fix-delete-by-filter-es
Signed-off-by: Slimane AMAR <[email protected]>
…hub.com/gridsuite/network-modification-server into fix-delete-by-filter-es
…ion-server into fix-delete-by-filter-es
|
On a filter deletion with a lot of things to delete, ES returns a "All shards have failed" error.
That's because we are asking ElasticSearch to return all the equipment infos corresponding to deleted identifiables ids. If we have a lot of equipments deleted, the following list:
deletedEquipments.stream().map(EquipmentInfosToDelete::id).toList(),
can contain more than 10000 elements.
This equipment infos list is then used to avoid sending delete requests to ES for equipments that don't exist in the index. This is not necessary as ElasticSearch can handle this directly. In terms of performance, it's more efficient to send delete request with documents that don't exist instead of fetching the documents and deleting only the existing ones from my tests.
I added another constant for deletion partitioning as the existing one is too big for deletions. If there are too many ids in the request, the number of clauses is too large and can't be handled by ElasticSearch. The error looks like:
on my local deployment.
From the documentation, in the version we use there is a heuristic to determine the maxClauseCount from the available memory/heap (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-settings.html).
I set it by default to 2048 to be below the limit in our local deployment but we could override this value in other environment if we're looking to improve performance of delete with a lot of values.