Scraper fixes #1137

ewan-escience · 2024-03-01T11:49:26Z

Scraper fixes

Changes proposed in this pull request:

Fix bug where an OpenAlex raw author was wrongly assumed to be not null
Adapt to the changed DataCite GraphQL API, where the publisher now has sub fields
Escape colons when searching in the DataCite GraphQL API on title
Show DataCite instead of Crossref when DataCite was used as the source in bulk import
Set the container names for Swagger and CodeMeta in production

How to test:

docker compose down --volumes && docker compose build --parallel && docker compose up --scale data-generation=0
Create a software or project page, publish it
Add the following DataCite mentions directly, then remove them, then add them with bulk import:
- 10.48550/arxiv.2310.12084
- 10.5281/zenodo.1140396
Search for the title CellProfiler and KNIME: Open-Source Tools for High-Content Screening, a DataCite result should be shown, add it.
Run the mentions scraper: docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainMentions
The scraper should run without errors, the scraped_at fields should be set at http://localhost/api/v1/mention
Add the following reference papers:
- 10.1021/acs.jpclett.9b01634
- 10.1021/ja026939x
Run the citation scraper: docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainCitations, it should run without errors