Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrapers failing because of github requests being rate limited #1300

Open
mlandauer opened this issue Jul 14, 2022 · 1 comment
Open

scrapers failing because of github requests being rate limited #1300

mlandauer opened this issue Jul 14, 2022 · 1 comment

Comments

@mlandauer
Copy link
Member

This is happening every so often with various repositories. For example:

Octokit::TooManyRequests: GET https://api.github.com/repos/OddBloke/ontario_cannabis_store_scraper/contributors: 403 - API rate limit exceeded for 173.255.208.251. (But here's the good news: Authentica...

(This is from logs in retrying background jobs in sidekiq)

What's weird is according to the code it looks like the api request for contributors is being done via an authenticated request where the user is one associated with the scraper (so most likely the owner)

Hmmm... This will need to be investigated because it does seem to be causing trouble.

@mlandauer
Copy link
Member Author

Could it be that for some reason some users' authentication tokens aren't valid, those users are trying requests enough that our IP is being rate limited irrespective of whether other calls are being made in an authenticated way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant