Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Goal: Reduce Network Usage #764

Open
1 task
MicBun opened this issue Dec 12, 2024 · 2 comments
Open
1 task

Goal: Reduce Network Usage #764

MicBun opened this issue Dec 12, 2024 · 2 comments

Comments

@MicBun
Copy link
Contributor

MicBun commented Dec 12, 2024

detected our network usage was incredibly high (TB/day), and it was increasing proportionally with amount of streams
requests seemed to be coming from KGW since its outgoing data was low while incoming was high (eth0 interface)
Determined that the source is KGW polling all the schemas
We'll temporarily make this poll interval to 5 minutes. Consequence: we'll probably get dataset/schema/procedure not found errors right after creating streams. It only affects deployment. If desired, we can target nodes directly to avoid this error while deploying

Making the network usage lower, will reduce our cost on infrastructure

Problem:

@zolotokrylin
Copy link
Contributor

what's the benefit here? can you quantify it please?

@MicBun
Copy link
Contributor Author

MicBun commented Dec 18, 2024

what's the benefit here? can you quantify it please?

I edited the Goal description, the benefit is lower network usage and lowering cost of our infrastructure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants