Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reaperscans #2277

Merged
merged 7 commits into from
Feb 25, 2024
Merged

fix: reaperscans #2277

merged 7 commits into from
Feb 25, 2024

Conversation

kuwoyuki
Copy link
Contributor

@kuwoyuki kuwoyuki commented Feb 22, 2024

some cancer laravel framework and ratelimits
closes #1702

Copy link
Owner

@dipu-bd dipu-bd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require clarification

@kuwoyuki
Copy link
Contributor Author

kuwoyuki commented Feb 23, 2024

done but resolve_futures doesn't seem to work properly with LOG_LEVEL=DEBUG. And the executor ratelimit dosen't really work nicely with this source, it does finish eventually with retries tho.

Used this novel for testing with 20 pages/637 chapters.

@dipu-bd
Copy link
Owner

dipu-bd commented Feb 25, 2024

For LOG_LEVEL=DEBUG, it just does not show the progress bar.

I had trouble scraping from reaperscans before as well. one day it works, the other day requests fail frequently. we can test using different ratelimits for this. e.g. a value of 0.5 might fare better.

I will merge this commit as it is now. If you have found a good ratelimit for reaperscans please make another PR

@dipu-bd dipu-bd merged commit abc97fa into dipu-bd:dev Feb 25, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants