Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate limits should be definable by extraction-worker pool #36

Closed
TimDaub opened this issue Jul 11, 2022 · 1 comment
Closed

Rate limits should be definable by extraction-worker pool #36

TimDaub opened this issue Jul 11, 2022 · 1 comment

Comments

@TimDaub
Copy link
Collaborator

TimDaub commented Jul 11, 2022

  • many cloud providers have exact rate limits usually denoted in calls/seconds, e.g. https://docs.infura.io/infura/networks/ipfs/how-to/request-rate-limits "You can retrieve IPFS data using the API with a limit of 100 requests per second for the remaining read-only methods such as:"
  • But don't confuse this with extraction-worker's concurrency which just says: independent of time, there should always been 100 connections open (we call this concurrency).
  • While concurrency is important and we should keep it, another useful option would be to allow users to define rate limit quotas, e.g. "1/s" (1 call per second) or "10/ms" (10 calls per millisecond).
  • It'd be ideal if a strategy implementer would add those quotas as property to a worker message: https://github.com/neume-network/message-schema/blob/a66dbf12a37709177aa238b2c8e08ec105a90097/src/index.mjs#L140
  • and then extraction-worker would queue those calls respectively such that they never go over the defined quota, e.g. extraction-worker would guarantee that many messages with a quota: "1/s" would never exceed 1 call per second etc.
@TimDaub
Copy link
Collaborator Author

TimDaub commented Aug 6, 2022

closed in #38

@TimDaub TimDaub closed this as completed Aug 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant