Non-default min_free_kbytes

[gaurntnx]

ZFS assumes min_free_kbytes to be the default as computed by Linux. There are situations where someone may need to bump up min_free_kbytes to a larger value. In such a case arc_sys_free will still be very low compared to the Zone LOW_WMARK and ARC may not even try to evict buffers unless kswap kicks in.

For e.g if min_free_kbytes is set to 512000 (500MB), the LOW_WMARK for NORMAL zone is around 580MB..After boost this value may go as high as 1.58 GB. arc_sys_free will be computed based on default Linux logic and will be around 1.4G for a 40GB system (37MB * 3 + 3% of mem) . So in this case, even though the free memory is below LOW_MARK, ARC wouldnt proactively do anything.

Key point: Why are we assuming that min_free_kbytes wont deviate from the default. And if it does, how do we best handle it? One option could be to introduce a module param that can be passed the right value. Although this wouldnt take care of situations when min_free_kbytes is changed during runtime. The other alternative is to just override the value to arc_sys_free and bypass the ZFS computation? Any thoughts/suggestions.