Merge to master: Release 2.4.0 - Multi-backend stores integrations, U…

…RI content negotiation and Ontology metadata and data indexation (#73)

* Feature: Migrate to virtuoso (#67)

* set up multiple triple store test environment

* optimize api tests

* Fix: update ncbo_annotator gem version (#71)

* update ncbo_annotator gem version

* update alegrograph to version 8.1.0

* Feature: use the new  SOLR Schema API instead of SOLR config files  (#68)

* update docker compose to use standard SOLR not the ontoportal configured

* update term search to use the new Schema API and remove config files

* update properties search to use the new Schema API & remove config files

* update class and properties schema to use the existent dynamic names

* Feature: resolving resources within specific ontologies, supporting various output formats(#69)

* remove useless line preventing sending the reset password email (#65)

* [ontoportal-bot] Gemfile.lock update

* Feature: api endpoint returns json-ld  for the element with that URI

* implement GET, POST requests, and GET /parse to submit INRATHES ontology

* Enhance tests using real data submission

* Enhance bin/ontoportal to make it able to run localy with UI

* Small fixes

- change controller name and test controller name
- remove /parse endpoint
- rackup to shotgun in bin/ontoportal

* Fix test dereference resource controller

- in json test, before we test the result we sort the hashes with the function (sort_nested_hash)
- in xml, ntriples and turtle, we split the result and the expected result, sort them and compare them

* update gemfile: add json-ld (3.0.2)

* change derefrencement namespacing and clean code

* Fix dereference resource tests expected resultsto handle parse triples

* fix xml serialization test for AG and Gb by cleaning the xml string

---------

Co-authored-by: OntoPortal Bot <[email protected]>
Co-authored-by: imadbourouche <[email protected]>

* Feature: URI drerfrencement content negotiation (#72)

* remove useless line preventing sending the reset password email (#65)

* [ontoportal-bot] Gemfile.lock update

* Feature: api endpoint returns json-ld  for the element with that URI

* implement GET, POST requests, and GET /parse to submit INRATHES ontology

* Enhance tests using real data submission

* Enhance bin/ontoportal to make it able to run localy with UI

* Small fixes

- change controller name and test controller name
- remove /parse endpoint
- rackup to shotgun in bin/ontoportal

* Fix test dereference resource controller

- in json test, before we test the result we sort the hashes with the function (sort_nested_hash)
- in xml, ntriples and turtle, we split the result and the expected result, sort them and compare them

* update gemfile: add json-ld (3.0.2)

* change derefrencement namespacing and clean code

* Fix dereference resource tests expected resultsto handle parse triples

* Feature: add content negotiation middleware

* Add headers to tests instead of output_format

* Apply middleware to only /ontologies/:acronym/resolve/:uri

* Add test cases for AllegroGraph and fix xml test

* move the content_negotiation middleware into rack folder and  module

* re-implement again the usage of  the output_format param if no format is given in the request header

* clean the tests for no more necessary checks

* clean and simplify the content negotiation middleware

* add the accepted format in the error response of resolvability endpoint

* refactor the content negotiation middleware code to be more clear

---------

Co-authored-by: Syphax bouazzouni <[email protected]>
Co-authored-by: OntoPortal Bot <[email protected]>

* Fix: user creation security  (#60)

* extract slice tests helper to the parent class for reusability

* add a test for the creation of an admin user

* enforce the security of admin user creation

* update slices controller to enforce admin security

* Fix: the content negotiation by removing a no needed require

* Feature: Indexation administration & Ontologies and Agents search  (#70)

* index submission and agents metadata

* add search administration endpoints to init schema and index batch

* add ontology and agent search endpoints

* add agent and ontology search tests

* add admin search in collections

* make the search admin use directly the solr connector

* implement search ontologies content search endpoint

* enforce solr models indexing one by one to prevent batch fails

* add detType tov search ontologies  content search endpoint

* fix content ontology search pagination

* add ontology search content types filter

* Feature: Add accessibility security to ontology metadata & content search results  (#74)

* add ontology accessibility restriction  to ontology metadata search

* add ontology accessibility restriction  to ontology content search

* add search results accessibility security test

* fix: enable user creation notification (#76)

* Fix: Invalidating cache on insert & fix Redis warning (#77)

* Merge pull request https://github.com/ncbo/ontologies_api/pull/120from ncbo/remove_redis-activesupport

Remove redis activesupport

* use the branch development of sparql client

* Feature: mappings statistics slices support (#78)

* restrict mapping statistics ontologies to the ontologies of the current slice

* add a test for the mappings slices support

* add test for mappings statistics slices support

---------

Co-authored-by: OntoPortal Bot <[email protected]>
Co-authored-by: imadbourouche <[email protected]>
Co-authored-by: Bilel Kihal <[email protected]>

.github/workflows/ruby-unit-tests.yml

@@ -7,25 +7,35 @@ on:
 jobs:
   test:
     strategy:
+      fail-fast: false
       matrix:
-        backend: ['api'] # api runs tests with 4store backend and api-agraph runs with AllegroGraph backend
+        goo-slice: [ '20', '100', '500' ]
+        ruby-version: [ '2.7' ]
+        triplestore: [ 'fs', 'ag', 'vo', 'gb' ]
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - name: copy-env-config
-      run: cp .env.sample .env
-    - name: Build docker-compose
-      run: docker-compose --profile 4store build #profile flag is set in order to build all containers in this step
-    - name: Run unit tests
-      # unit tests are run inside a container
-      # http://docs.codecov.io/docs/testing-with-docker
-      run: |
-        ci_env=`bash <(curl -s https://codecov.io/env)`
-        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle install 
-        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle exec rake test TESTOPTS='-v'
-    - name: Upload coverage reports to Codecov
-      uses: codecov/codecov-action@v3
-      with:
-        flags: unittests
-        verbose: true
-        fail_ci_if_error: false # optional (default = false)
+      - uses: actions/checkout@v3
+      - name: Install Dependencies
+        run: sudo apt-get update && sudo apt-get -y install raptor2-utils
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby-version }}
+          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
+      - name: Run unit tests
+        # unit tests are run inside a container
+        # http://docs.codecov.io/docs/testing-with-docker
+        run: |
+          ci_env=`bash <(curl -s https://codecov.io/env)`
+          GOO_SLICES=${{ matrix.goo-slice }} bundle exec rake test:docker:${{ matrix.triplestore }} TESTOPTS="-v"
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          flags: unittests
+          verbose: true
+          fail_ci_if_error: false # optional (default = false)

Dockerfile

@@ -7,6 +7,7 @@ RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends \
   openjdk-11-jre-headless \
   raptor2-utils \
   wait-for-it \
+  libraptor2-dev \
   && rm -rf /var/lib/apt/lists/*
 
 RUN mkdir -p /srv/ontoportal/ontologies_api

Gemfile

@@ -14,6 +14,9 @@ gem 'sinatra', '~> 1.0'
 gem 'sinatra-advanced-routes'
 gem 'sinatra-contrib', '~> 1.0'
 gem 'request_store'
+gem 'parallel'
+gem 'json-ld'
+
 
 # Rack middleware
 gem 'ffi'
@@ -27,9 +30,8 @@ gem 'rack-timeout'
 gem 'redis-rack-cache', '~> 2.0'
 
 # Data access (caching)
-gem 'redis', '~> 4.8.1'
-gem 'redis-activesupport'
-gem 'redis-store', '1.9.1'
+gem 'redis'
+gem 'redis-store', '~>1.10'
 
 # Monitoring
 gem 'cube-ruby', require: 'cube'
@@ -44,12 +46,12 @@ gem 'haml', '~> 5.2.2' # pin see https://github.com/ncbo/ontologies_api/pull/107
 gem 'redcarpet'
 
 # NCBO gems (can be from a local dev path or from rubygems/git)
-gem 'goo', git: 'https://github.com/ontoportal-lirmm/goo.git', branch: 'master'
-gem 'ncbo_annotator', git: 'https://github.com/ontoportal-lirmm/ncbo_annotator.git', branch: 'master'
+gem 'ncbo_annotator', git: 'https://github.com/ontoportal-lirmm/ncbo_annotator.git', branch: 'development'
 gem 'ncbo_cron', git: 'https://github.com/ontoportal-lirmm/ncbo_cron.git', branch: 'master'
 gem 'ncbo_ontology_recommender', git: 'https://github.com/ncbo/ncbo_ontology_recommender.git', branch: 'master'
-gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
-gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'master'
+gem 'goo', github: 'ontoportal-lirmm/goo', branch: 'development'
+gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'development'
+gem 'ontologies_linked_data', git: 'https://github.com/ontoportal-lirmm/ontologies_linked_data.git', branch: 'development'
 
 group :development do
   # bcrypt_pbkdf and ed35519 is required for capistrano deployments when using ed25519 keys; see https://github.com/miloserdow/capistrano-deploy/issues/42
@@ -74,5 +76,5 @@ group :test do
   gem 'rack-test'
   gem 'simplecov', require: false
   gem 'simplecov-cobertura' # for codecov.io
-  gem 'webmock'
+  gem 'webmock', '~> 3.19.1'
 end