You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
So, I was using this code (and thank you for making it) to convert a read Kafka stream into batches (to reduce concurrent load on the downstream destination) -
The issue is that it seems like some ReadableStreams don't actually ever invoke flush, making the strictMode irrelevant
as a replacement, i added a "maxWait" parameter that records the timestamp of the last update, and upon receiving a put into the BatchStream, checks whether the required batch size has been reached OR the maxWait has elapsed since the last put.
This way, there is an occasional internal "flushing" of the stream without waiting for an externally invoked flush...
wdyt?
The text was updated successfully, but these errors were encountered:
So, I was using this code (and thank you for making it) to convert a read Kafka stream into batches (to reduce concurrent load on the downstream destination) -
The issue is that it seems like some ReadableStreams don't actually ever invoke flush, making the strictMode irrelevant
as a replacement, i added a "maxWait" parameter that records the timestamp of the last update, and upon receiving a put into the BatchStream, checks whether the required batch size has been reached OR the maxWait has elapsed since the last put.
This way, there is an occasional internal "flushing" of the stream without waiting for an externally invoked flush...
wdyt?
The text was updated successfully, but these errors were encountered: