Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1117 disable scraping #1153

Merged
merged 2 commits into from
Mar 29, 2024
Merged

1117 disable scraping #1153

merged 2 commits into from
Mar 29, 2024

Conversation

ewan-escience
Copy link
Collaborator

Disallow Git scraping

Changes proposed in this pull request:

  • Admins can disable the scraping of a Git repo, maintainers can see the reason why

How to test:

  • docker compose down --volumes && docker compose build --parallel && docker compose up --scale data-generation=0
  • Sign in as admin, create two software pages, set a repo for both, then fill the disallow scraping reason in for one of them
  • Wait for the scrapers to run
  • After a few minutes, the page without a reason should be scraped, the other one should not
  • On the background services tab, you should be able to see a reason, both as admin and maintainer
  • On the links & metadata tab, the edit reason field should only be visible for admins

Related to #1117 (we still need to do this for package managers)

PR Checklist:

  • Increase version numbers in docker-compose.yml
  • Link to a GitHub issue
  • Update documentation
  • Tests

@@ -128,6 +128,17 @@ export default function AutosaveRepositoryUrl() {
}
}, [urlError, repository_url, platform.id])

async function saveScrapingDisabledReason({value}: any) {
return fetch(`/api/v1/repository_url?software=eq.${id}`, {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please handle the error in try/catch and show it to using showErrorMessage. The value should not be any but a string|null, right?

@ewan-escience ewan-escience force-pushed the 1117-disable-scraping branch from e499d1c to 65ab5ed Compare March 29, 2024 09:23
Copy link
Contributor

@dmijatovic dmijatovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!

Copy link

Quality Gate Passed Quality Gate passed for 'rsd-database'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

Copy link

Copy link

@ewan-escience ewan-escience merged commit 8459fe1 into main Mar 29, 2024
8 checks passed
@ewan-escience ewan-escience deleted the 1117-disable-scraping branch April 12, 2024 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants