Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: scrape organisation location from ror #1127

Merged
merged 5 commits into from
Feb 23, 2024

Conversation

cmeessen
Copy link
Contributor

@cmeessen cmeessen commented Feb 22, 2024

Adds ROR scraper for organisation's city and country

Fixes #806

Changes proposed in this pull request:

  • adds a new scraper that collects country and city from ROR and stores them in the database
  • the scraper takes organisations that do have a ROR id, but do not have a country OR a city and overwrites both information with data from ROR

How to test:

  • optional: add MAX_REQUESTS_ROR to .env
  • docker compose down --volumes && docker compose build --parallel && docker compose up
  • login, create a new software and add a couple of organisations
  • wait a few minutes for the scraper to run or run:
    docker compose exec scrapers /opt/java/openjdk/bin/java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.ror.MainRor
    
  • verify that countries appear inside cards of the organisation overview

PR Checklist:

  • Increase version numbers in docker-compose.yml
  • Link to a GitHub issue
  • Update documentation
  • Tests

@cmeessen cmeessen force-pushed the 806-scrape-organisation-location branch 4 times, most recently from 4db9e85 to 819c161 Compare February 22, 2024 13:14
@cmeessen cmeessen force-pushed the 806-scrape-organisation-location branch from 819c161 to ade237d Compare February 22, 2024 13:17
@cmeessen cmeessen marked this pull request as ready for review February 22, 2024 13:19
@cmeessen cmeessen self-assigned this Feb 22, 2024
@ewan-escience ewan-escience force-pushed the 806-scrape-organisation-location branch from e75b370 to 9d0713a Compare February 22, 2024 15:17
@ewan-escience ewan-escience force-pushed the 806-scrape-organisation-location branch from 9d0713a to 7bc755b Compare February 23, 2024 09:30
@ewan-escience ewan-escience force-pushed the 806-scrape-organisation-location branch from 45575d8 to f1d7e2b Compare February 23, 2024 10:25
Copy link
Contributor

@dmijatovic dmijatovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

One suggestion. Maybe we can add slug of the organisation to link to organisation page directly from error logs (admin section), see 118-backend-logs-views.sql RPC script. This can be done later too.

Copy link

Quality Gate Passed Quality Gate passed for 'rsd-database'

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

Copy link

@dmijatovic
Copy link
Contributor

One suggestion. Maybe we can add slug of the organisation to link to organisation page directly from error logs (admin section), see 118-backend-logs-views.sql RPC script. This can be done later too.

I added this feature in the last commit.

@cmeessen cmeessen merged commit 7bdab6a into main Feb 23, 2024
6 checks passed
@dmijatovic dmijatovic deleted the 806-scrape-organisation-location branch March 21, 2024 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scrape organization location
3 participants