Add Solr gem to DGU Find

[kentsanggds]

Add Solr Gem to find as a feature so can switch between Opensearch and Solr

Tasks

Beta Give feedback

1. Get Solr data from Integration
2. Sanitize search input
3. Compare results to OpenSearch
4. Add tests
5. Fix feature flag routing
6. Investigate indexing Publisher, Format and Topics in ckanext-datagovuk to be run as a cronjob

Search options

Beta Give feedback

1. User input - to check sanitization etc
2. Publisher
3. Topic
4. Format
5. OGL
6. Results ranking - best match
7. Results ranking - most recent
8. Add tests

Search results page

Beta Give feedback

1. Title
2. Organisation - "extras_dcat_publisher_name" is not always available, "organizations" returns slug
3. Last updated
4. Summary
5. Filter by publisher (currently returns "organizations" slugs taken from CKAN API /organization_list endpoint, to look into Solr facet and aggregation for distinct values
6. Filter by topic (currently hardcoded as this seems definitive)
7. Filter by format (currently hardcoded as this seems definitive)
8. Filter by OGL
9. Clicking on a result link goes to dataset page (to point to /solr)
10. Availability (shows "not released" if dataset does not have datafiles)
11. Search results pagination
12. Add tests
13. Sort results by best match (default sort option)
14. Sort results by most recent - sort using public_updated_at and order desc
15. Update Publisher to use solr index
16. Update Topic to use solr index
17. Update Format to use solr index

Individual dataset page

Beta Give feedback

1. Breadcrumb - to add /search and look into @referer_query and @referer
2. Header Metadata
3. Metadata box - availability, license_id(unpublished,other,ogl,other-at should display text "None"), topic (currently slug) left
4. Link to show more datasets from publisher
5. Search box
6. Get data from "location" field and how it is used in the view(?) - to confirm it's not being used
7. List data links (non time series files) if present
8. List data links (timeseries data files?) if present - groups files by year, haven't found example of this yet. Looks like it was used for legacy datasets so we won't implement this for now (to add to a decision log)
9. Show supporting docs if available
10. Show additional information if present
11. Show contact details if publisher has contact email - inconsistent data for contact-email and foi-email, manually passed to organisation model?
12. Show licence information (review raw custom licence / other )
13. Show option to login to edit CKAN dataset if has not been harvested
14. Return a 404 if the dataset is empty or does not exist
15. Option to get dataset by legacy name?
16. Show related datasets if present - part of search functionality, see https://solr.apache.org/guide/8_11/morelikethis.html
17. Add tests
18. Datafile (datafile["created"]) is not always available

Preview dataset page

Beta Give feedback

1. Organogram preview (not loading)
2. CSV preview

Pull requests

Beta Give feedback

1. Add Solr and basic search page #1313
   +0
2. Set Solr URL in Find govuk-dgu-charts#309
   +0
3. Add sort filters to search page #1314
   +0
4. https://github.com/alphagov/datagovuk_find/tree/add-basic-dataset-page
5. https://github.com/alphagov/datagovuk_find/tree/add-pagination-controls

[kentsanggds]

We had a chat about how to proceed with progressing and reviewing the work in smaller chunks.

Proposal is to create another branch main-solr from which other solr update branches will be fed into. This will ensure that smaller PRs can be reviewed and merged into it without affecting any deployments to production.

The inteegration cluster can be updated to point to commits on this branch specifically when there is a need to deploy things for testing on EKS.

As there is generally very little development on the Find application I don't forsee any issues around the branch drifting from the main branch.