Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix read capacity calculation and wait between scans #63

Closed
wants to merge 1 commit into from

Conversation

peterjuras
Copy link

Related to #60

The current implementation of the usable read capacity has some issues and might lead to provisioned throughput exceptions.

The current implementation has two problems:

  • All items are treated as units of 8kb, although some items might be smaller or larger than 8kb. This leads to problems especially where items are larger than 8kb.
  • The current read capacity limit is effectively ignored, since the limit has a small impact on a single request, but there is no waiting period between requests. Therefore, the current implementation is solely bound by the available resources (most likely network) and is theoretically able to perform a full table backup within one second. Provisioned read capacity is however limited to a single second, therefore the new implementation will now wait until enough capacity is available again to proceed to the next scan request.

The new implementation solves these issues and will no longer exceed the requested limit. Exceptions to this case are tables that have a very low provisioned read capacity (e.g. 1), where the percentage limit has little effect and will result in an average consumption of 1 read capacity.

Backups might now take longer, but at least they respect the given read capacity.

@peterjuras
Copy link
Author

Closed due to inactivity of repository.

@peterjuras peterjuras closed this Jun 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant