S3V2: Increase memory overhead ratio

[johnny-schmidt]

What

This seems to keep the big avro/parquet syncs from grinding to a halt on high-work streams.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 17, 2024 7:54pm

[tryangul]

Do we want to make the default higher in general?

Also is this mixing concerns? E.g. are we indirectly trying to account for the memory usage by the writer (process records) in the input queues?

[johnny-schmidt]

Do we want to make the default higher in general?

Also is this mixing concerns? E.g. are we indirectly trying to account for the memory usage by the writer (process records) in the input queues?

I mean, we're trying to get the message queues to respect the contract that they only use approximately 20% of memory. (Tho arguably that does mix concerns.

Maybe these message queues just need to be aggressively limited, like if there's no work to do downstream we're just wasting effort clearing stdin.