[BUG] Cache maximum size settings don't work as expected when pluggable caching is on

[peteralfonsi]

Describe the bug

Cache implementations have settings which are used to set the max size. For example, OpenSearchOnHeapCache has indices.requests.opensearch_onheap.size. These settings should override the generic IndicesRequestCache size setting indices.requests.cache.size.

But, after some recent changes, these settings aren't having any effect. This is happening because in the TieredSpilloverCache, there are smaller segments which can each have their own instance of OpenSearchOnHeapCache, and the TSC passes in the per-segment max size to the cache config used to build each OpenSearchOnHeapCache. To support this, OpenSearchOnHeapCache (and other cache impls) were changed to override its own setting value with the value passed in through the config.

This is fine for TSC segments. But, if we're just using OpenSearchOnHeapCache, the IndicesRequestCache also puts its default 1% size value into the cache config, and indices.requests.opensearch_onheap.size is incorrectly ignored.

This is only an issue if the pluggable caching feature flag is on.

Related component

Search

To Reproduce

Start a cluster with pluggable caching on and the max size setting set to some value: ./gradlew run -Dtests.opensearch.opensearch.experimental.feature.pluggable.caching.enabled=true -Dtests.opensearch.indices.requests.cache.store.name=opensearch_onheap -Dtests.opensearch.indices.requests.cache.opensearch_onheap.size=40MB --debug-jvm

Check in the IntelliJ debugger that the actual max size of the OpenSearchOnHeapCache object is set to 1% of the heap size instead of 40 MB

Expected behavior

The max size should match the setting value.

Additional Details

No response