Indexing: N/A values should not be indexed

[pdurbin]

"I noticed that in #1246 that we don't want to display N/A values in UI, but as of this writing it's easy to create a dataset like this via SWORD by simply not including any dcterms:subject elements in the XML"

I first made that comment a month ago at #1430 (comment) but I'm opening this ticket (with a screenshot) to make sure we're ok with showing N/A in the Subject facet:

This happens all the time on https://apitest.dataverse.org right now where I'm not specifying a subject in the XML I use to create a dataset via SWORD.

[scolapasta]

We should not be indexing these N/A values.

[scolapasta]

Notice with production data we also see several N/As.

[pdurbin]

We should not be indexing these N/A values.

@scolapasta please review a195b17 (pushed to a branch) and let me know if it's ok to merge in. The question I have is for the dataset level. In that commit the logic is to never index an "N/A" value for any dataset field that has a controlled vocabulary that includes "N/A" (not just the "subject" field).

[pdurbin]

I chatted with @scolapasta and it sounds like we want to avoid indexing "N/A" for as many fields as possible, including fields that don't have controlled vocabularies (which is most fields).

Since the logic I'll need to put in the code is spread on various lines, I just did a query on our migrated data to make sure I adjust indexing for as many fields as possible. The main ones to adjust will be for authors and descriptions:

[pdurbin@dvn-vm7 ~]$ curl -s 'http://localhost:8983/solr/collection1/select?rows=1000000&wt=json&indent=true&q="N/A"' | grep '"N/A"' | sort | uniq -c | sort -nr
   6563         "dsDescriptionValue":["N/A"],
   6062         "authorName_ss":["N/A"],
   6062         "authorName":["N/A"],
     12         "producerURL":["N/A"],
      6         "universe_ss":["N/A"],
      6         "universe":["N/A"],
      4         "relatedMaterial":["N/A"],
      4         "otherGeographicCoverage":["N/A"],
      3         "relatedDatasets":["N/A"],
      2         "gsdSiteType_ss":["N/A"],
      2         "gsdSiteType":["N/A"],
      2         "geographicUnit_ss":["N/A"],
      2         "geographicUnit":["N/A"],
      2         "dvDescription":"N/A",
      2         "description":"N/A",
      1         "title":"N/A",
      1         "publicationCitation":["N/A"],
      1         "nameSort":"N/A",
      1         "gsdProgramBrief_ss":["N/A"],
      1         "gsdProgramBrief":["N/A"],

[pdurbin]

@scolapasta please review the new-and-improved version at 269081b and let me know if I should merge it to master.

[pdurbin]

@scolapasta merged #1897 but we decided to change break to continue in 50a4b9a so that we don't stop indexing values on the first "N/A" in a list of controlled vocabulary values. Some of those values may not be "N/A".

Ready for QA.

[posixeleni]

@scolapasta @kcondon I cant test this in build since we need to see how migrated local studies got indexed. So can you please tell me when this fix is in the new production site (harvard.dataverse.edu) or vm6?

cc/ @sbarbosadataverse

[posixeleni]

Searched for N/A in production and was unable to find it indexed. Also looked at facets to confirm this was the case. Closing this ticket.