Merge pull request IQSS#10579 from GlobalDataverseCommunityConsortium…

…/change_solr_doc_deletes

Solr: don't delete docs that will just change

doc/release-notes/10579-avoid-solr-deletes.md

@@ -0,0 +1,9 @@
+A features flag called "reduce-solr-deletes" has been added to improve how datafiles are indexed. When the flag is enabled, 
+Dataverse wil avoid pre-emptively deleting existing solr documents for the files prior to sending updated information. This
+should improve performance and will allow additional optimizations going forward.
+
+The /api/admin/index/status and /api/admin/index/clear-orphans calls
+(see https://guides.dataverse.org/en/latest/admin/solr-search-index.html#index-and-database-consistency)
+will now find and remove (respectively) additional permissions related solr documents that were not being detected before.
+Reducing the overall number of documents will improve solr performance and large sites may wish to periodically call the 
+clear-orphans API.

doc/sphinx-guides/source/developers/performance.rst

@@ -121,6 +121,7 @@ While in the past Solr performance hasn't been much of a concern, in recent year
 We are tracking performance problems in `#10469 <https://github.com/IQSS/dataverse/issues/10469>`_.
 
 In a meeting with a Solr expert on 2024-05-10 we were advised to avoid joins as much as possible. (It was acknowledged that many Solr users make use of joins because they have to, like we do, to keep some documents private.) Toward that end we have added two feature flags called ``avoid-expensive-solr-join`` and ``add-publicobject-solr-field`` as explained under :ref:`feature-flags`. It was confirmed experimentally that performing the join on all the public objects (published collections, datasets and files), i.e., the bulk of the content in the search index, was indeed very expensive, especially on a large instance the size of the IQSS prod. archive, especially under indexing load. We confirmed that it was in fact unnecessary and were able to replace it with a boolean field directly in the indexed documents, which is achieved by the two feature flags above. However, as of writing this, this mechanism should still be considered experimental.
+Another flag, ``reduce-solr-deletes``, avoids deleting solr documents for files in a dataset prior to sending updates. It also eliminates several causes of orphan permission documents. This is expected to improve indexing performance to some extent and is a step towards avoiding unnecessary updates (i.e. when a doc would not change).
 
 Datasets with Large Numbers of Files or Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/sphinx-guides/source/installation/config.rst

@@ -3274,6 +3274,9 @@ please find all known feature flags below. Any of these flags can be activated u
     * - add-publicobject-solr-field
       - Adds an extra boolean field `PublicObject_b:true` for public content (published Collections, Datasets and Files). Once reindexed with these fields, we can rely on it to remove a very expensive Solr join on all such documents in Solr queries, significantly improving overall performance (by enabling the feature flag above, `avoid-expensive-solr-join`). These two flags are separate so that an instance can reindex their holdings before enabling the optimization in searches, thus avoiding having their public objects temporarily disappear from search results while the reindexing is in progress. 
       - ``Off``
+    * - reduce-solr-deletes
+      - Avoids deleting and recreating solr documents for dataset files when reindexing. 
+      - ``Off``
 
 **Note:** Feature flags can be set via any `supported MicroProfile Config API source`_, e.g. the environment variable
 ``DATAVERSE_FEATURE_XXX`` (e.g. ``DATAVERSE_FEATURE_API_SESSION_AUTH=1``). These environment variables can be set in your shell before starting Payara. If you are using :doc:`Docker for development </container/dev-usage>`, you can set them in the `docker compose <https://docs.docker.com/compose/environment-variables/set-environment-variables/>`_ file.

src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java

@@ -1,5 +1,6 @@
 package edu.harvard.iq.dataverse;
 
+import edu.harvard.iq.dataverse.DatasetVersion.VersionState;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
 import edu.harvard.iq.dataverse.dataaccess.DataAccess;
 import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter;
@@ -383,7 +384,8 @@ public FileMetadata findMostRecentVersionFileIsIn(DataFile file) {
         if (fileMetadatas == null || fileMetadatas.isEmpty()) {
             return null;
         } else {
-            return fileMetadatas.get(0);
+            // This assumes the order of filemetadatas is from first to most recent, which is true as of v6.3 
+            return fileMetadatas.get(fileMetadatas.size() - 1);
         }
     }
 
@@ -759,6 +761,13 @@ public List<DataFile> findAll() {
         return em.createQuery("select object(o) from DataFile as o order by o.id", DataFile.class).getResultList();
     }
 
+    public List<VersionState> findVersionStates(Long fileId) {
+        Query query = em.createQuery(
+                "select distinct dv.versionState from DatasetVersion dv where dv.id in (select fm.datasetVersion.id from FileMetadata fm where fm.dataFile.id=:fileId)");
+        query.setParameter("fileId", fileId);
+        return query.getResultList();
+    }
+
     public DataFile save(DataFile dataFile) {
 
         if (dataFile.isMergeable()) {