Problem: High network usage in instances

[outerlook]

TL DR
KGW is polling our whole data too frequently to determine which streams exist. It will be temporarily configured to have a lower frequency (default 6 seconds) while the Kwil team works on solving this issue. --schema-sync-interval will be set to 300 secs

Process for future reference

• detected our network usage was incredibly high (TB/day), and it was increasing proportionally with amount of streams
• requests seemed to be coming from KGW since its outgoing data was low while incoming was high (eth0 interface)
• Determined that the source is KGW polling all the schemas
• We'll temporarily make this poll interval to 5 minutes. Consequence: we'll probably get dataset/schema/procedure not found errors right after creating streams. It only affects deployment. If desired, we can target nodes directly to avoid this error while deploying

cc @MicBun

[outerlook]

another temporary solution we could take if a lower frequency turns to be bad: moving all nodes to same AZ in AWS. At least cost/GB is free

[MicBun]

We'll temporarily make this poll interval to 5 minutes.

I think more than 5 minutes is still good, most of our data are updated daily and we don't need a fast sync yet.

[zolotokrylin]

Is it a goal or a bug?

[outerlook]

Is it a goal or a bug?

significative limitation? adding to https://github.com/trufnetwork/truf-network/issues/90 was good