Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a blocking task executor so that we can easily bound the number of tasks kept live in memory at once #2136

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

poodlewars
Copy link
Collaborator

Apply it to the finalize_staged_data process as a first example.

There are simpler ways to implement this in do_compact specifically (like completing a future before submitting a new one), and other places in the code use folly::window to do this, but this gives us a good way to handle these problems in a cross-cutting way. We might want to use the blocking queue more widely if this is successful.

This helped to reduce the memory used by finalize_staged_data with write delays, I'm going to do some more benchmarking then share the results here.

Monday: 8234437666

…f tasks kept live in memory at once.

Apply it to the finalize_staged_data process as a first example.

Monday: 8234437666
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant