Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Enhance SOLR integration and add a Schema API #54

Merged
merged 25 commits into from
Mar 2, 2024
Merged
Changes from 1 commit
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
b16ffbd
add an abstraction to SOLR integeration and add Schema API
syphax-bouazzouni Feb 3, 2024
0389943
add SOLR Schema API tests
syphax-bouazzouni Feb 3, 2024
f3815c4
update SOLR backend configuration and init
syphax-bouazzouni Feb 3, 2024
9ca2e1c
use the new Solr connector in the model search interface
syphax-bouazzouni Feb 3, 2024
459c4ff
update search test to cover the new automatic indexing and unindexing
syphax-bouazzouni Feb 3, 2024
bde2bdf
handle the solr container initialization when running docker for tests
syphax-bouazzouni Feb 3, 2024
df31c26
add omit_norms options for SolrSchemaGenerator
syphax-bouazzouni Feb 3, 2024
adcf79d
fix solr schema initial dynamic fields declaration and replace the us…
syphax-bouazzouni Feb 5, 2024
67023bf
delay the schema generation to after model declarations or in demand
syphax-bouazzouni Feb 5, 2024
bccf3e7
add solr edismax fitlers tests
syphax-bouazzouni Feb 5, 2024
d4f5e1d
fix indexBatch and unindexBatch tests
syphax-bouazzouni Feb 5, 2024
8308441
add security checks to the index and unindex functions
syphax-bouazzouni Feb 5, 2024
877bbe1
change dynamic fields names to have less code migration
syphax-bouazzouni Feb 9, 2024
9d5c23d
update clear_all_schema to remove all copy and normal fields
syphax-bouazzouni Feb 9, 2024
23a0824
add an option to force solr initialization if wanted
syphax-bouazzouni Feb 16, 2024
664127c
handle indexing embed objects of a model
syphax-bouazzouni Feb 9, 2024
6a29f05
add index update option
syphax-bouazzouni Feb 16, 2024
d00f9a5
fix clear all schema to just remove all the fields and recreate them
syphax-bouazzouni Feb 16, 2024
ec99ed8
add index_enabled? helper for models
syphax-bouazzouni Feb 16, 2024
ca32e8c
perform a status test when initializing the solr connector
syphax-bouazzouni Feb 17, 2024
83ac6f6
Merge branch 'development' into feature/add-model-based-search
syphax-bouazzouni Feb 23, 2024
e461c2d
extract init_search_connection function from init_search_connections
syphax-bouazzouni Feb 24, 2024
51fbfe4
fix typo in indexOptimize call
syphax-bouazzouni Feb 25, 2024
10b90c1
Merge branch 'development' into feature/add-model-based-search
syphax-bouazzouni Feb 29, 2024
84aa3db
add solr search using HTTP post instead of GET for large queries
syphax-bouazzouni Mar 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
handle indexing embed objects of a model
syphax-bouazzouni committed Feb 16, 2024
commit 664127cdf7cb0a3773c4d5c7382fb9b2ba3d2a2c
3 changes: 3 additions & 0 deletions lib/goo.rb
Original file line number Diff line number Diff line change
@@ -291,6 +291,9 @@ def self.add_search_connection(collection_name, search_backend = :main, &block)
block: block_given? ? block : nil
}
end
def self.search_connections
@@search_connection
end

def self.init_search_connections(force=false)
@@search_collections.each do |collection_name, backend|
32 changes: 30 additions & 2 deletions lib/goo/search/search.rb
Original file line number Diff line number Diff line change
@@ -37,11 +37,39 @@ def index_doc(to_set = nil)
raise NoMethodError, "You must define method index_doc in your class for it to be indexable"
end

def embedded_doc
raise NoMethodError, "You must define method embedded_doc in your class for it to be indexable"
end

def indexable_object(to_set = nil)
begin
document = index_doc(to_set)
rescue
document = self.to_hash.reject { |k, _| !self.class.indexable?(k) }
document = self.to_hash.reject { |k, _| !self.class.indexable?(k) }
document.transform_values! do |v|
is_array = v.is_a?(Array)
v = Array(v).map do |x|
if x.is_a?(Goo::Base::Resource)
x.embedded_doc rescue x.id.to_s
else
if x.is_a?(RDF::URI)
x.to_s
else
x.respond_to?(:object) ? x.object : x
end
end
end
is_array ? v : v.first
end

document = document.reduce({}) do |h, (k, v)|
if v.is_a?(Hash)
v.each { |k2, v2| h["#{k}_#{k2}".to_sym] = v2 }
else
h[k] = v
end
h
end
end

model_name = self.class.model_name.to_s.downcase
@@ -91,7 +119,7 @@ def schema_generator
end

def index_document_attr(key)
return key.to_s if custom_schema?
return key.to_s if custom_schema? || self.attribute_settings(key).nil?

type = self.datatype(key)
is_list = self.list?(key)
11 changes: 9 additions & 2 deletions test/test_search.rb
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@
module TestSearch

class TermSearch < Goo::Base::Resource
model :term_search, name_with: :id
model :term_search, name_with: lambda { |resource| uuid_uri_generator(resource) }
attribute :prefLabel, enforce: [:existence]
attribute :synonym, enforce: [:list] # array of strings
attribute :definition # array of strings
@@ -83,6 +83,10 @@ class TermSearch3 < Goo::Base::Resource
attribute :semanticType
attribute :cui

attribute :object, enforce: [:term_search]
attribute :object_list, enforce: [:term_search, :list]


enable_indexing(:test_solr)
end

@@ -332,7 +336,10 @@ def test_index_on_save_delete
definition: "definition of test2",
synonym: ["synonym1", "synonym2"],
submissionAcronym: "test",
private: true
private: true,
object: TermSearch.new(prefLabel: "test", submissionAcronym: 'acronym', submissionId: 1 ).save,
object_list: [TermSearch.new(prefLabel: "test2",submissionAcronym: 'acronym2', submissionId: 2).save,
TermSearch.new(prefLabel: "test3", submissionAcronym: 'acronym3', submissionId: 3).save]
)

term.save