desc = series has too many labels

[omartheironman]

Hello,

on one on our distributors im seeing a rise in CPU and exponential rise n memory. Initially I thought the memory being occupied by something eventually exhausting the CPU leading to 100%.

When i saw the logs I see distributor with multiple 4xx
desc = series has too many labels

I was wondering what happens to these requests to they get piled up somewhere?

[friedrichg]

if ingesters are slow to respond, distributors will hold more requests, causing them to OOM
You can prevent some distributor OOMing setting up a limit to inflight pushes

  # Max inflight push requests that this distributor can handle. This limit is
  # per-distributor, not per-tenant. Additional requests will be rejected. 0 =
  # unlimited.
  # CLI flag: -distributor.instance-limits.max-inflight-push-requests
  [max_inflight_push_requests: <int> | default = 0]

https://cortexmetrics.io/docs/configuration/configuration-file/#distributor_config