Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement job queue and thread pool for I/O to files and plugins #121

Open
aremmell opened this issue Jul 13, 2023 · 2 comments
Open

Implement job queue and thread pool for I/O to files and plugins #121

aremmell opened this issue Jul 13, 2023 · 2 comments
Assignees
Labels
enhancement like a new feature, just not as cool

Comments

@aremmell
Copy link
Owner

If a write is to a network drive, or a plugin makes a network call as part of its behavior, the thread that called into libsir could stall.

Instead of letting that happen, stick the write job into a queue that is picked up by a pool of threads which do nothing except wait for a job to be available, process it, and resume waiting.

@aremmell aremmell added the enhancement like a new feature, just not as cool label Jul 13, 2023
@aremmell aremmell self-assigned this Jul 13, 2023
@aremmell
Copy link
Owner Author

Implemented in thread-pool branch. More testing and consideration must go into it before putting it to use. As of now, not even sure it is a solution to the original issue:

A thread queue with 4 threads calling into a plugin or network file that blocks for 60 seconds will immediately be fully occupied - and therefore:

  • needlessly growing the queue of incoming jobs in RAM
  • will never catch up, because the log messages will continue to come in faster than the messages can be dispatched, even by 4 separate runners.

Perhaps we should stash away output meant for destinations we know to currently be blocking/taking too long to write, then when their responsiveness is better, pull it out from a file and write it to the dest.

@johnsonjh
Copy link
Collaborator

This should also be used for syslog and systemd logging. The possibility of blocking here is low but real.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement like a new feature, just not as cool
Projects
None yet
Development

No branches or pull requests

2 participants