Scraper for RSD aggregator

[ewan-escience]

Scraper for RSD aggregator

Changes proposed in this pull request

• Some small changes in the new database structure
• Add the scraper for aggregating RSDs

How to test

• docker compose down --volumes && docker compose build --parallel && docker compose up --scale data-generation=1
• Run the following in the database: INSERT INTO remote_rsd(label, "domain") VALUES ('RSD', 'https://research-software-directory.org/') ,('NFDI', 'https://helmholtz.software/') ,('local', 'http://nginx/');
• Note that we added localhost so we can also test the delete functionality
• Wait for the scraper to run (should we within a minute) or run docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.aggregator.MainAggregator
• Check if http://localhost/api/v1/remote_software has the expected results
• Check if http://localhost/api/v1/rpc/aggregated_software_overview has the expected results
• Sign in as admin, delete or unpublish a published software page, wait 5 minutes or clear the column scraped_at of the localhost entry: UPDATE remote_rsd SET scraped_at = NULL WHERE "label" = 'local';. The entry should be deleted.
• There should be no errors of this scraper in the error logs or in the Docker output.

Partially implements #1287

PR Checklist:

• Increase version numbers in docker-compose.yml
• Link to a GitHub issue
• Update documentation
• Tests

[sonarqubecloud]

Quality Gate passed for 'rsd-database'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
13.2% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'scrapers'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[dmijatovic]

Nice work!