Store Gateway: Cache expanded postings for expensive matcher

[yeya24]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Store Gateway Index Cache caches each posting individually. For example, pod="prometheus-0" is one cache key and pod="prometheus-1" can be another cache key. This is kind of problematic for those high cardinality labels for example, host, node or pod.

When there is a query matcher such as pod!="" , Store Gateway tries to get all values of label pod and fetches each value from the remote cache. This is ok for small set up. However, we have 100+ vertically sharded blocks per day where each block has close to 1M values for pod label. When a query tries to fetch 30 days of data with matcher pod!="", this ended up fetching 3B cache keys from remote cache. It is even worse when there is cache miss for postings, Store Gateway tries to backfill remote cache with postings data fetched from object store. This can be another 3B SET requests to the cache.

Lazy posting is a way to solve high cardinality matchers such as pod!="" and we have several attempts to improve that. However, the root cause of the problem is that Index Cache stores each label (posting) separately and it is not doing well against high cardinality labels.

Changes

This PR made several changes:

• For each posting group (most of the cases this is a single label matcher in your query), if it matches more than 1 key, fetch expanded postings from cache for this group. For example, pod!="" usually matches more than 1 pod so it tries to fetch expanded postings matching pod!="".
• If cache miss, Store Gateway fallbacks to fetch individual postings for pod from cache and object storage.
• If Store Gateway downloads a posting from object store, if the posting's corresponding posting group matches more than 1 key, Store Gateway doesn't backfill remote cache, instead it tries to backfill the whole expanded posting for the whole posting group
• For each posting group which fetches only a single key such as __name__="metric_name", it works the same as of today.

Basically we change from caching individual postings to caching the whole matcher.

now:
pod=A => [1,2,3,4]
pod=B => [5,6,7,8]
pod=C => [9, 10]

new:
pod!="" => [1,2,3,4,5,6,7,8,9,10]

Index Cache interface FetchExpandedPostings is updated to support multiple matchers. The previous one is designed to fetch only one set of matchers.

today: FetchExpandedPostings(ctx context.Context, blockID ulid.ULID, matchers []*labels.Matcher, tenant string) ([]byte, bool)

new: FetchExpandedPostings(ctx context.Context, blockID ulid.ULID, matchers [][]*labels.Matcher, tenant string) [][]byte

Verification

Tests updated.

[yeya24]

Maybe we can make the condition > a certain number instead of > 1

[yeya24]

In case expanded posting cache miss, we still try to fetch each posting individually from index cache and then object store.

I wonder if we should just skip cache and fetch from object store directly. It makes no sense to fetch postings from cache if we don't store cache there