Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: add readme for scrapers and add code coverage badges for severa… #1271

Merged
merged 1 commit into from
Aug 14, 2024
Merged

docs: add readme for scrapers and add code coverage badges for severa… #1271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions authentication/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ SPDX-License-Identifier: CC-BY-4.0

# Authentication module

[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=nl.research-software%3Aauthentication&metric=coverage)](https://sonarcloud.io/summary/overall?id=nl.research-software%3Aauthentication)

This module handles authentication from third parties using oAuth2 and OpenID.

## Environment variables
Expand Down
2 changes: 2 additions & 0 deletions codemeta/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ SPDX-License-Identifier: CC-BY-4.0

# CodeMeta server

[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=rsd-codemeta&metric=coverage)](https://sonarcloud.io/summary/overall?id=rsd-codemeta)

This module implements a [CodeMeta](https://codemeta.github.io/) server, using [V3](https://w3id.org/codemeta/v3.0) of the schema.

If your instance of the RSD is hosted on https://example.com/, you can access the CodeMeta server on https://example.com/codemeta/v3/. To get the CodeMeta data for a software page, just append its slug to this URL. For example, to get the CodeMeta data for software page on https://example.com/software/some-software, visit https://example.com/codemeta/v3/some-software/.
Expand Down
2 changes: 2 additions & 0 deletions frontend/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ SPDX-License-Identifier: CC-BY-4.0

# Research Software Directory (RSD) - Frontend

[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=rsd-frontend&metric=coverage)](https://sonarcloud.io/summary/overall?id=rsd-frontend)

Based on the features in the legacy application and the current requirements we selected [Next.js](https://nextjs.org/docs) and MUI-5 (https://mui.com/getting-started/usage/) frameworks for:

- Easy integration with generic SSO oAuth services like ORCID, SURFconext, Microsoft etc.
Expand Down
32 changes: 32 additions & 0 deletions scrapers/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
<!--
SPDX-FileCopyrightText: 2024 Ewan Cahen (Netherlands eScience Center) <[email protected]>
SPDX-FileCopyrightText: 2024 Netherlands eScience Center

SPDX-License-Identifier: CC-BY-4.0
-->

# Scrapers

[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=nl.research-software%3Ascrapers&metric=coverage)](https://sonarcloud.io/summary/overall?id=nl.research-software%3Ascrapers)

This module contains the scrapers used by the RSD. The scrapers are written in Java using Maven.

## Running from within Docker Compose

If you have an instance of the RSD, including the scrapers, running with Docker Compose, you can manually run a scraper by running the respective command from the root of the project content:

```shell
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainProgrammingLanguages
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainContributors
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainBasicData
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainCommits
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainMentions
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainReleases
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.package_manager.MainPackageManager
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.doi.MainCitations
docker compose exec scrapers java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.ror.MainRor
```

## Running locally

To run a scraper locally, e.g. from an IDE, you need to make sure the values in `.env` are loaded and that the value for the env variable `POSTGREST_URL` is overwritten with an address that is reachable from outside of the Docker network (usually `http://localhost/api/v1`).
Loading