Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update ElasticSearch and Elastica to 6.x #1880

Merged
merged 35 commits into from
Dec 5, 2024
Merged

Update ElasticSearch and Elastica to 6.x #1880

merged 35 commits into from
Dec 5, 2024

Conversation

anvit
Copy link
Contributor

@anvit anvit commented Oct 9, 2024 •
edited
Loading

Update ElasticSearch and Elastica and address issues introduced by breaking changes in ElasticSearch 6.x.

The most notable changes are:

  • Dropped support for multiple mapping types: ES 6.x no longer allows using multiple indices, They suggest handling this by creating multiple indices, each with its own type.
  • Mapping changes: The include_in_all mapping parameter is disallowed and the _all field is disabled by default.

Changes in Elastica:

  • Elastica only has a addDocuments, deleteDocuments, and updateDocuments methods in 6.x and there are no longer addDocument, deleteDocument, updateDocument methods for handling a singular documents.

@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 9f05a24 to 1e9114f Compare October 10, 2024 00:48
@anvit anvit self-assigned this Oct 10, 2024
@anvit anvit requested a review from a team October 10, 2024 16:22
@anvit anvit added this to the 2.9.0 milestone Oct 10, 2024
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 1e9114f to e6a7473 Compare October 10, 2024 22:11
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 7bdf937 to 0842eb9 Compare October 21, 2024 23:49
jraddaoui
jraddaoui requested changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great @anvit!

I have not been able to look at everything in detail yet, but I have added a few comments that I think are important and may require some work already. Let me know your thoughts ;)

apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated Show resolved Hide resolved
lib/QubitLftSyncer.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchIndexDecorator.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPluginUtil.class.php Outdated Show resolved Hide resolved
anvit and others added 10 commits November 15, 2024 15:53
@anvit
Update ElasticSearch and Elastica dependencies
81617d6 
Update ElasticSearch to 6.8.23 in the Dockerfile and update Elastica to
6.x in composer.
@melaniekung @anvit
Remove arElasticSearchIndexDecorator.
7c50884 
- 'inline' scripts are deprecated, changed to 'source'
@anvit
Update index for ES 6.x
1c5feeb 
Update arElasticSearchPlugin to use multiple indices instead of multiple
types since ES 6.x removed being able to add multiple types. Also update
mapping to remove include_in_all and add that to the _all field using
copy_to instead since include_in_all was also removed in ES 6.x
@anvit
Fix autocompleteAction for ES 6.x
ec4e748 
Update autocompleteAction to use multiple indices
@anvit
Add index types for ES Queries without a type
a19f8e1 
Add the appropriate index types for ES requests that did not need to
specify explicit types in ES 5.x
@anvit
Fix updateByQuery calls for ElasticSearch 6.x
90d1e08
@anvit
Fix broken publication status update
47cc1a9 
Add missing document type to partialUpdate in arElasticSearchPlugin
@anvit
Fix linting errors
3adaac0
@anvit
Switch elasticsearch to the oss docker image
5575f96
@anvit
Update mapping condition for _all fields
0259ed0 
Update condition in arElasticSearchPluginUtil to only get string
fields included in _all if copy_to is explicitly set to _all in the
mappping.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from abb1768 to 0259ed0 Compare November 15, 2024 23:53
anvit added 3 commits November 18, 2024 13:35
@anvit
Update ES mapping to copy relevant fields to _all
a13821d
@anvit
Rename arElasticSearchIndexDecorator
3f0e4d6 
Renamed arElasticSearchIndexDecorator as
arElasticSearchMultiIndexWrapper for clarity and added a few comments
about changes needed for ElasticSearch/Elastica 7.x.
@anvit
Rename getType function from ESMultiIndexWrapper
0e0d87d 
Rename getType from arElasticSearchMultiIndexWrapper to getIndex for
clarity.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from bbf4bb1 to 0e0d87d Compare November 19, 2024 00:09
@anvit anvit requested a review from jraddaoui November 19, 2024 00:18
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from a37039a to 4b52458 Compare November 19, 2024 00:49
@anvit
Add dummy type for ES index
bc54543 
Add a dummy ElasticSearch index type since one is still required for
ES 6.x, instead of having a seperate type name for each of the
multiple indices.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from 4b52458 to bc54543 Compare November 19, 2024 01:02
jraddaoui
jraddaoui reviewed Nov 20, 2024
View reviewed changes
Copy link
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anvit! This looks a lot clearer to me now.

plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
@anvit
Change references to index type for ElasticSearch
9d6fa03 
Update any references to ElasticSearch index type to index name.
Also add getType as an alias for getIndex for backwards compatibility
with custom themes that still might be making references to
ElasticSearch's getType.
anvit added 2 commits November 21, 2024 16:56
@anvit
Remove unnecessary ElasticSearch type variables
9adb350 
Remove unnecessary $type ElasticSearch variables in QubitLftSyncer,
arElasticSearchPlugin, and updatePublicationStatusTask
@anvit
Remove ES index prefix from MultiIndexWrapper
fc6d069 
Remove the index prefix from arElasticSearchMultiIndexWrapper and use
AtoM class names internally instead of using prefix to refer to them.
jraddaoui
jraddaoui reviewed Nov 22, 2024
View reviewed changes
Copy link
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anvit! Getting there, just a few minor tweaks!

apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated Show resolved Hide resolved
lib/QubitLftSyncer.class.php Outdated Show resolved Hide resolved
lib/QubitLftSyncer.class.php Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchMultiIndexWrapper.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Outdated Show resolved Hide resolved
anvit added 3 commits November 22, 2024 12:04
@anvit
Fix autocompleteAction bug, address CR feedback
c6f0c2a 
Fix bug in search autocompleteAction which was showing duplicate results
for actors under repositories. Also address some of the CR feedback and
update/add methods for add, update, and deleteById in the
arElasticSearchMultiIndexWrapper.
@anvit
Remove non text fields from _all ES mapping
9859544
@anvit
Remove unnecessary loop in arElasticSearchPlugin
ef14555 
Remove unnecessary foreach loop in arElasticSearchPlugin. Also move code
for adding filter for stripping md tags into a seperate method to make
the initialization code more readable.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from c5bbe4a to ef14555 Compare November 22, 2024 20:48
jraddaoui
jraddaoui reviewed Nov 25, 2024
View reviewed changes
Copy link
Contributor

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks @anvit!

apps/qubit/modules/search/actions/autocompleteAction.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Outdated Show resolved Hide resolved
plugins/arElasticSearchPlugin/lib/arElasticSearchPlugin.class.php Show resolved Hide resolved
plugins/arElasticSearchPlugin/config/mapping.yml Show resolved Hide resolved
@anvit
Update ES index refresh to only refresh one index
02b3c4b 
Update the index refresh method to accept an optional name param that
only refreshes that particular index in
arElasticSearchMultiIndexWrapper. Also address some small CR feedback.
@anvit anvit force-pushed the dev/upgrade-elasticsearch branch from a94554a to 02b3c4b Compare November 25, 2024 22:49
@melaniekung melaniekung merged commit 9f553d3 into qa/2.x Dec 5, 2024
6 checks passed
@anvit anvit linked an issue Dec 6, 2024 that may be closed by this pull request
Problem: ElasticSearch 5.x dependency #1708
Closed
@anvit anvit deleted the dev/upgrade-elasticsearch branch December 10, 2024 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jraddaoui jraddaoui jraddaoui approved these changes

Assignees

@anvit anvit

Labels
None yet
Projects
None yet
Milestone
2.9.0
Development

Successfully merging this pull request may close these issues.

Problem: ElasticSearch 5.x dependency
3 participants
@anvit @jraddaoui @melaniekung