Feature: Enhance SOLR integration and add a Schema API

[syphax-bouazzouni]

Prerequisites

• Feature: Add after_save and after_destroy hooks to models  #53
• SOLR >= 8.11 and started in mode cloud

Goals

• Make it easier to configure SOLR programmatically.
• Have the possibility to index any model on demand

Context

SOLR is the indexing tool, that we use for our search features, it works by defining a collection (a table in the databases world), and for each collection, a schema defines the properties to index by giving its type, a list, or not, ... and also some dynamic or special fields to handle fuzzy search, or other.

The requirement was that we were required to define the collection and the schema, in XML configuration files at the start, and then after we could not change it in the code. This meant we were limited in the action that we could do, and it was hard to add new search features to our system. as these files were static, and you had to update the schema and create the collection configuration files each time you wanted to add something into the index.

This PR, integration the SOLR Schema API, in this project, gives us the option to create/delete a collection and update a collection schema dynamically, the following actions were implemented (see the full list in the SOLR::Admin, SOLR::Schema and SOLR::SchemaGenerator modules in the code):

• SOLR Administration API

    fetch_all_collections 
    
    create_collection(name = @collection_name, num_shards = 1, replication_factor = 1)
   
    delete_collection(collection_name = @collection_name)
    
    collection_exists?(collection_name)

• SOLR Schema API

    add_field(name, type, indexed: true, stored: true, multi_valued: false)
    
    add_dynamic_field(name, type, indexed: true, stored: true, multi_valued: false)
    
    add_copy_field(source, dest)
    
    add_field_type(type_definition)

    delete_field(name)

    update_schema(schema_json)
   
    fetch_schema

In Addition to the implementation of the SOLR Schema and admin APIs, we added a dsl to the Goo model, to enable index for any model, either in a schemaless mode or in a custom schema model

• Schemaless mode: This will generate a collection and schema using the settings/metadata of the model properties, it is uses SOLR dynamic fields to make this possible, below an example of how to do it.

  class TermSearch < Goo::Base::Resource
    model :term_search, name_with: :prefLabel
    attribute :prefLabel, enforce: [:existence], fuzzy_search: true # fuzzy search will permit to have autocomplete for this field 
    attribute :synonym, enforce: [:list]
    attribute :definition
    attribute :submissionAcronym, enforce: [:existence]
    attribute :submissionId, enforce: [:existence, :integer]
    attribute :private, enforce: [:boolean], default: false, index: false # this field not be indexed 
    attribute :semanticType
    attribute :cui

    enable_indexing(:my_collection) # will generate the collection called `my_collection` and index all the attributes of the model (except `:private` here)
  end

• Custom Schema mode: This we generate a collection and a schema directly defined in the code and decelartion of the model, below an example

class TermSearch < Goo::Base::Resource
    model :term_search, name_with: :id
    attribute :prefLabel, enforce: [:existence]
    attribute :synonym, enforce: [:list] # array of strings
    attribute :definition  # array of strings
    attribute :submissionAcronym, enforce: [:existence]
    attribute :submissionId, enforce: [:existence, :integer]
    attribute :semanticType
    attribute :cui

    enable_indexing(:term_search) do
      schema_generator.add_field(:prefLabel, 'text_general', indexed: true, stored: true, multi_valued: false)
      schema_generator.add_field(:synonym, 'text_general', indexed: true, stored: true, multi_valued: true)
      schema_generator.add_field(:definition, 'string', indexed: true, stored: true, multi_valued: true)
      schema_generator.add_field(:submissionAcronym, 'string', indexed: true, stored: true, multi_valued: false)
      schema_generator.add_field(:submissionId, 'pint', indexed: true, stored: true, multi_valued: false)
      schema_generator.add_field(:cui, 'text_general', indexed: true, stored: true, multi_valued: true)
      schema_generator.add_field(:semanticType, 'text_general', indexed: true, stored: true, multi_valued: true)

  
      # Copy fields for fuzzy and autocomplete search
      schema_generator.add_copy_field('prefLabel', '_text_')
      schema_generator.add_copy_field('prefLabel', 'prefLabel_Exact')
      schema_generator.add_copy_field('prefLabel', 'prefLabel_Suggest')
      schema_generator.add_copy_field('prefLabel', 'prefLabel_SuggestEdge')
      schema_generator.add_copy_field('prefLabel', 'prefLabel_SuggestNgram')

      schema_generator.add_copy_field('synonym', '_text_')
      schema_generator.add_copy_field('synonym', 'synonym_Exact')
      schema_generator.add_copy_field('synonym', 'synonym_Suggest')
      schema_generator.add_copy_field('synonym', 'synonym_SuggestEdge')
      schema_generator.add_copy_field('synonym', 'synonym_SuggestNgram')

      schema_generator.add_copy_field('notation', '_text_')
    end
end 

Changes

• add an abstraction to SOLR integration and add Schema API(b16ffbd)
• add SOLR Schema API tests(0389943)
• update SOLR backend configuration and init(f3815c4)
• use the new Solr connector in the model search interface(9ca2e1c)
• update search test to cover the new automatic indexing and unindexing (459c4ff)

[codecov]

Codecov Report

Attention: Patch coverage is 81.01604% with 71 lines in your changes are missing coverage. Please review.

Project coverage is 86.36%. Comparing base (1be1c83) to head (84aa3db).

Files	Patch %	Lines
lib/goo/search/solr/solr_schema.rb	72.34%	26 Missing ⚠️
lib/goo/search/search.rb	82.79%	16 Missing ⚠️
lib/goo/search/solr/solr_admin.rb	69.76%	13 Missing ⚠️
lib/goo/search/solr/solr_query.rb	87.50%	6 Missing ⚠️
lib/goo/search/solr/solr_schema_generator.rb	85.36%	6 Missing ⚠️
lib/goo.rb	92.30%	2 Missing ⚠️
lib/goo/search/solr/solr_connector.rb	91.66%	2 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff               @@
##           development      #54      +/-   ##
===============================================
+ Coverage        85.93%   86.36%   +0.43%     
===============================================
  Files               41       46       +5     
  Lines             2702     3022     +320     
===============================================
+ Hits              2322     2610     +288     
- Misses             380      412      +32     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mdorf]

@syphax-bouazzouni, this looks great! Really useful feature for us as well! It would be great to test it against the latest Solr 9.5.0 to make sure the code is compatible. I am wrapping up my work on a few search enhancements for the RADx project, which we plan to deploy to production shortly. The next major task is upgrading Solr to the latest version, which we may very well coincide with merging this pull request. Are you planning to make any other significant changes to this feature (you mentioned that this is a "first iteration)? Thank you!

[syphax-bouazzouni]

9.5.0

Hello, @mdorf, I tested it locally with Solr 9.5.0 and all the tests are green.
I removed the "first iteration" term, as I implemented everything I had in mind for now: administrate SOLR in code, be backward compatible with the existent, and index models automatically at save/delete.

[mdorf]

@syphax-bouazzouni, I am working on merging this functionality into our develop branch. It's a bit tricky given that the pull request is not against our own repo. Probably will need to do a lot of manual merging. Were you planning on submitting this pull request against the ncbo repo? If so, should I wait for that or just proceed with my manual merging? Thank you!

[jonquet]

Hello, the proposition was to move and create PRs directly on the OntoPortal repo now that everyone is positioned under it. I think @syphax-bouazzouni has scheduled some time to create PR related to our work soon.

[syphax-bouazzouni]

@syphax-bouazzouni, I am working on merging this functionality into our develop branch. It's a bit tricky given that the pull request is not against our own repo. Probably will need to do a lot of manual merging. Were you planning on submitting this pull request against the ncbo repo? If so, should I wait for that or just proceed with my manual merging? Thank you!

Hello @mdorf, I would suggest waiting at least 1 month before merging this, as it is only tested in our development environment, and will be released in our next release, see ontoportal-lirmm/ontologies_api#73

Once deployed to our production environment and tested, I will do a PR.

Is it good with you?

[mdorf]

@syphax-bouazzouni, @jonquet, no problem. In general, we are very interested in this feature to facilitate the functionality sought by the RADx project, in which our Solr index would be packaged in a way to be accessible by the third-party API. See bmir-radx/radx-project#49. However, based on my conversation with @alexskr, this does not have the immediate urgency. A month is definitely reasonable for us to wait to be able to merge this feature in its more stable and tested iteration.