[Feature Request] Add setting controlling whether to allow size > 0 requests in the request cache

[peteralfonsi]

Is your feature request related to a problem? Please describe

Currently size > 0 requests aren't cached by default. This is because they usually don't have aggregations, and weren't considered worth storing in the limited on-heap request cache. But, with the rollout of tiered caching, the request cache can be much larger, and it can be worthwhile to cache size > 0 requests in this case. We expect this should increase request cache utilization, and therefore also help with latency.

Currently, the user can already explicitly allow size > 0 queries to be cached, by passing the URL parameter request_cache=true in the request. But, users may not be aware of this, and it might not be feasible for them to manually make this change.

Describe the solution you'd like

We should expose a dynamic cluster setting controlling whether size > 0 requests are cacheable. The default would be false, reflecting the current behavior. Such requests would still have to meet the other criteria for cacheability (no DFS queries, no now queries, etc.)

Related component

Search:Performance

Describe alternatives you've considered

In the future we may also want a more flexible/generic caching policy system for the request cache, where checking for cacheability is delegated to the cache implementations themselves. This would allow us to choose based on time took, size, or anything else that might be relevant to individual cache implementations.

However this would be significantly more complex and we think the most gain can come from allowing size > 0 requests to be cached.

Additional context

No response

[dbwiddis]

Thanks for explaining the need for this! I like that the default doesn't change anything but it enables opt-in caching.